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Abstract. We use the uniqueness of various invariant functionals on irreducible unitary representations 
of PGZ/2(R) in order to deduce the classical Rankin-Selberg identity for the sum of Fourier coefficients 
of Maass cusp forms and its new anisotropic analog. We deduce from these formulas non-trivial bounds 
for the corresponding unipotent and spherical Fourier coefficients of Maass forms. As an application we 
obtain a subconvexity bound for certain L-functions. Our main tool is the notion of a Gelfand pair from 
the representation theory. 



1. Introduction 

In this paper we study periods of automorphic functions. We present a new method which allows one 
to obtain non-trivial spectral identities for weighted sums of certain periods of automorphic functions. 
These identities are modelled on the classical identity of R. Rankin |Ra| and A. Selberg [Se]. We recall 
that the Rankin-Selberg identity relates the weighted sum of Fourier coefficients of a cusp form tj> to the 
weighted integral of the inner product of </> 2 with the Eisenstein series (e.g., formula ()1.7j) below). 

We show how to deduce the classical Rankin-Selberg identity and similar new identities from the 
uniqueness principle in representation theory (also known under the following names: the multiplicity 
one property, Gelfand pair). The uniqueness principle is a powerful tool in representation theory; it plays 
an important role in the theory of automorphic functions. 

We associate a non-trivial spectral identity to certain pairs of different triples of Gelfand subgroups. 
Namely, we associate a spectral identity (see the formula (|1.4[) below) with two triples T C Hi C Q and 
T C Hi C Q of subgroups in a group Q such that pairs (Q,Hi) and (Hi, J 7 ) for i = 1, 2, are strong 
Gelfand pairs having the same subgroup T in the intersection (for the notion of Gelfand pair that we 
use, see Section fl.l.3|) . We call such a collection (G,Hi,H2,J r ) a strong Gelfand formation. 

In the reminder of the Introduction we explain our general idea and describe how to implement it in 
order to reprove the classical Rankin-Selberg formula. We also obtain a new anisotropic analog of the 
Rankin-Selberg formula. We present then an analytical applications of these spectral identities towards 
non-trivial bounds for various Fourier coefficients of cusp forms. The novelty of our results lies mainly 
in the method, as we do not rely on the well-known technique of Rankin and Selberg which is called 
"unfolding" . Instead, we use the uniqueness of relevant invariant functionals which we explain below. 
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We would like to mention that the Rankin-Selberg method was revolutionized by H. Jacquet, J. Shalika 
and I. Piatetski-Shapiro who constructed integral representations for many automorphic L-functions (see 
the survey paper |Bu| and references therein). It is not yet clear what is the relation between these works 
and our point of view. 

1.1. Periods and Gelfand pairs. We briefly review some well-known notions and constructions from 
representation theory and the theory of automorphic functions needed to formulate our identities. 

Taking periods is the classical technique to study automorphic functions. It goes back, at least, to 
Hecke and Maass. In the classical language it means the following. Let 4> be an automorphic function on 
Y and let ip be an automorphic function on a cycle C C Y, where the cycle C is equipped with a measure 
dc. Then one can consider the period defined via the integral 

/ </)(c)'i/'(c) dc . 
Jc 

It is well-known that in the modern language of automorphic representations this construction leads to 
the following setup which we are going to use throughout the paper. 

1.1.1. Automorphic representations. Let Q be a real reductive Lie group (e.g., <SX2(R)), let Tg C Q be a 
lattice and let Xg = Tg \ Q be the corresponding automorphic space equipped with a (/-invariant measure 
jig (which we will always normalize to have the total mass one). We denote by L 2 [Xg) = L 2 (Xg, /ig) 
the corresponding unitary representation of Q and by C°°{Xg) its smooth part. 

Let (-7T, Q, V) be an abstract unitary irreducible representation of Q. We will work solely with the spaces 
of smooth vectors. Hence by (ir, V) we mean the representation in the space of smooth vectors equipped 
with the invariant Hermitian form. (Usually one denotes a unitary representation by (ir,Q,L) where L 
is the corresponding Hilbert space and the space of smooth vectors is denoted by V = L°°.) 

A pair (tv, v), where v : V — > L 2 (Xg) is an isometric t/-morphism, is called an automorphic representa- 
tion of Q on X g . It is known that v : V -> C°°{X g ) and we will denote by V£ ut = V {7r ,„) C C°°{Xg) the 
image of V under v. We denote by v* : C°°(Xg) — > V the adjoint map. 

1.1.2. Periods and representation theory. The notion of a period has the following counterpart in the 
language of automorphic representations. 

Let H C Q be a subgroup (not necessarily reductive, e.g., a unipotent subgroup of SL% (R)) and let 
Xji C Xg be a closed orbit of H (e.g., Xu = TL fl Tg \ H). We fix a W-invariant measure [iu on Xu and 
consider the unitary representation L 2 (X-h, Hn) of H. We denote by r« = rx n ■ C°°(Xg) — » C°°(Xft) 
the corresponding restriction map. 

Let (ir, u n ) be an automorphic representation of Q on Xg and let (cr, v a ) an automorphic representation 
of Ti. on Xyi . This means that for the abstract irreducible unitary representation a in the space of smooth 
vectors W equipped with the 7i-invariant Hermitian form, we fix an isometry v a : W — > L 2 (Xn); we 
allow ourselves to use the name automorphic even if Ti. is not reductive. Let v* : C°°(X'n) — > W be the 
adjoint map (induced by the 7i-invariant Hermitian form on L 2 {Xji, ^h))- 

We consider the Tt- equivariant map = Tj^ n a = v* o r-n o : V — » W defined via the composition 

T™ 4 : V ^ C°°{Xg) C°°{X H ) lz+ W . 

We call this map the automorphic period map (or simply the period) associated to the collection (w, v^), 
(cr, v a \ Xg, Xji and the choice of corresponding measures. This is the representation theoretic substitute 
for the classical period. It is well-defined if Xn is compact. Otherwise we have to assume that 7r or a 
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is cuspidal (i.e., rapidly decaying at infinity if Xji is not compact). For a non-cuspidal data one has to 
introduce appropriate regularization in order to make sense out of the automorphic period. 

Clearly T£"J € Hom w (V, W). We denote the vector space Hom w (V,W0 by V(V,W) and call it the 
vector space of periods between n and a. The space of periods is defined even if 7r or a are not automorphic. 

1.1.3. Gelfand pairs and periods. In many cases interesting periods are associated to some special sub- 
group (or representations). Many of these examples come from what is called the multiplicity one rep- 
resentations or Gelfand pairs (see [Gr] and references therein). In what follows we will use the notion of 
Gelfand pairs for real Lie groups. 

A pair (Q, Ti.) of a real Lie group Q and a real Lie subgroup Ti. C Q is called a strong Gelfand pair 
if for any pair of irreducible representations (it, V) of Q and (a, W) of H, the dimension of the space 
Hornby, W) of 7i-equivariant maps (i.e., the space of periods V(V, W)) for the spaces of smooth vectors 
is at most one. (It is well-known in representation theory of reductive groups that the dimension of 
the space Hom^V, W) for infinite- dimensional representations plays the role of the index for finite- 
dimensional representations.) The pair is called a Gelfand pair if the same holds for er being the trivial 
representation. 

One of the important observations in the theory of automorphic functions is that for Gelfand pairs, 
automorphic periods T%"£ lead to certain interesting numbers or functions (e.g., Fourier coefficients, 
L-functions). This is based on the fact that in many cases one can construct another vector in the 
one- dimensional vector space V(V, W). 

Namely, usually representations n and a could be constructed explicitly in some model spaces of sec- 
tions of various vector bundles over appropriate homogeneous manifolds (e.g., for SL% (ffi), in the spaces 
of homogeneous functions on the plane R 2 \0; see Section [2]). These models usually exist for all represen- 
tations and not only for the automorphic ones. If Am\V(V, W) = 1, using these models one can construct 
an explicit non-zero map T™° d £ V(V, W) by means of the corresponding kernel (i.e., construct this map 
as an integral operator with an explicit kernel). We call such a map a model period. 

When dim'P(V, W) = 1, such a choice of a non-zero model period T™° d gives rise to the automorphic 
coefficient of proportionality a, „ = a VitMa e C such that 



We would like to study these constants. In many cases these constants are related to interesting objects 
(e.g., Fourier coefficients of cusp forms, special values of L- functions etc.). Of course, these constants 
depend on the choice of model periods T™° d , normalization of measures, etc. In many cases we can find 
a way to canonically normalize norms of model maps in the adelic setting (and hence define canonically 
if not the constants themselves then their absolute values). We hope to discuss this question elsewhere. 

We explain now how sometimes one can obtain spectral identities involving certain coefficients o,^. 

1.1.4. Triples of Gelfand subgroups and periods. Let Q be a real Lie group and T C Q a real Lie subgroup 
which is not a Gelfand subgroup. Suppose we still want to study a period of an automorphic representation 
7r of G with respect to an ^"-invariant closed cycle Xjr C Xg (endowed with a JF-invariant measure fijr) 
and some automorphic representation x of T . For example, for the trivial representation of T we obtain 
the ^"-invariant functional Ij? on 7r given by the integral over the cycle Xf. 



We can not apply the idea described above directly. Instead in some cases we can obtain a spectral 
decomposition of this period. 



T 
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Namely, suppose we can find an intermediate subgroup TL, T C TL C Q, and an intermediate closed 
cycle Xjr c X n C Xg such that both pairs (Q, TL) and (TL, T) are strong Gelfand pairs. We claim that 
this leads to the spectral decomposition of the functional Ijr. In what follows we discuss for simplicity 
only the case of the trivial representation of T . The case of a non-trivial representation \ is similar (and 
leads to other interesting identities). 

Consider the space L 2 (X-j-c) = L 2 (Xji, ^h) and let assume that it has a decomposition into a direct 
sum (in general a direct integral) 

L 2 {X H ) = ® m (1.1) 

of irreducible automorphic representations (<Ji,v ai : Wi — > L 2 (Xfi)) of TL. This decomposition induces 
the spectral decomposition of the functional Ijr. 

In fact, the inclusion Xjr c X-n C Xg induces the period map I^ u *. £ V(V,C) for every <7j, via the 
following composition of maps 

I%% : V C°°(X g ) C°°(X H ) ^Wi^ C°°(X n ) C°°(A» C . 

The spectral decomposition (|1.1[) gives rise to the decomposition Ijr = (Since in order to 

compute Ijr(v) for a vector v G V, we first can restrict v„(v) to the cycle X^, decompose it with respect 
to the action of TL and then compute the integral over Xjr for each component.) Note that the functional 
1^. is the composition of two automorphic periods: 1^. — T£ ut c o T£ U J. : V — > W — > C. 

We now use the strong Gelfand property for the triple J 7 C TL C Q, i.e., that the product of dimensions 
dimHom w (T/, Wi) • dimHom^(Wi, C) < 1 for all W % . We choose model periods T™f G Hom w (y, Wi), 
jroixi g Homjp(Wi, C). As we explained above this leads to the automorphic coefficients of propor- 
tionality: T™J. = a n , ai ■ T™°f and Tffi. = 6 ffi)C • T™° c d . We denote by j Wt<Ti = ■ b a<JC and by 
= T™°£ o T™°f G V * . With such notations we arrive at the spectral decomposition of the functional 
Ijr g V* which is associated to the triple of strong Gelfand subgroups T C TL C Q 

Ir= E 7^-C:f- (1-2) 

Oi automoiphic 

Remark 1.1. We note that for a non-compact cycle Xjr there is no obvious way to write down the analog 
of the spectral decomposition (|1.2[) even if the initial representation n is cuspidal. This is because a priori 
the period T®™q might not be defined for all Oi (e.g., non-compact periods of Eisenstein series). Usually, 
one has to introduce an appropriate regularization procedure in order to define the corresponding periods. 
In this paper we only consider cycles Xjr which are compact and hence will not face this problem. 



1.2. Rankin- Selberg type spectral identities. Our main observation is that for a given pair of groups 
T C Q there might be different intermediate subgroups TL as above leading to different spectral decom- 
positions of the same functional Ijr and hence to identities between the automorphic coefficients. 

Let Q be a real Lie group and T C TLi C G, i = 1,2, be a collection of subgroups such that in the 
following commutative diagram each embedding is a strong Gelfand pair (i.e., pairs (G,TLi) and (TLi 7 J-) 
are strong Gelfand pairs) 

G (1-3) 




TL\ TL2 




T 
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We call such a collection of subgroups a strong Gelfand formation. 

Let r C Q be a lattice and ATg = V \ Q the corresponding automorphic space. Let X t = Xu t C 
and ATjr C Xg be closed orbits of Hi and J 7 , respectively, satisfying the following commutative diagram 
of embeddings 

Xg 




assumed to be compatible with the diagram (|1.3[) . We endow each orbit (as well as Xg) with a measure 
invariant under the corresponding subgroup (to explain the idea, we assume that all orbits are compact, 
and hence, these measures could be normalized to have mass one). 

We hx a decompositions L 2 (Xi) — (BiPi into a direct sum (in general into a direct integral) of auto- 
morphic representations (ci, v ai , Wi) of Hi and similarly l?{X-i) = ®jTj for automorphic representations 
(rj,v Tj ,Uj) of H 2 - 

Let (tt, v v ) be an automorphic representation of Q and : V — ► C the period defined by the integration 
over the cycle Xjr. As we explained in Section |j.l.4[ two different triples T C Hi C Q and T C H2 C Q 
lead to two spectral decompositions of the period I jr. Namely, let us choose the model periods T™° d E 
Hom Wl (V,Wi), T™c d G Hom^(Wi,C) and similarly T™° d E Homw 2 (V, f/j), T™° c d E Hom^([/ J ,C). This 
leads to the introduction of constants T% U J. = a n>(Ti T™f, T^ c = b Ui>c T^ and T%*£. = c V;Tj T^°f, 
T r"c = dr^cT^i as in Section [TX3J We denote by 7^ = aj' CT . • & CT(>C , 7™ J = T™° c d o T™f E y* and 
similarly by ^ = c^. • d Tj , c , I™f = T™° c d o T™° d E V*. The spectral decomposition (fOJ) for two 
triples of Gelfand pairs (| 1 ,3[) and the corresponding orbits f| 1 . 2[) implies the following identity 

E 7^ • /™°f = ^ = E <Wi • ^ • (1-4) 

We call such an identity the Rankin- Selberg type spectral identity or the period identity associated with 
the Gelfand formation (Q,Hi,H2 J 3 r ), the corresponding orbits and the automorphic representation tt. 
Note that the summation on the left in (|1.4p is over the set of irreducible representations of Hi occurring 
in L 2 (Xi) and the summation on the right is over the set of irreducible representations of H2 occurring 
in L 2 (X2). Since groups Hi and H2 might be quite different, the identity (|1.4|) is non-trivial in general. 
Surprisingly, even if Hi and H2 are conjugate in Q, the resulting identity is non-trivial in general (e.g., 
for Q = PGL2(M.) and two unipotent subgroups intersecting over T = e, this gives the Voronoi type 
summation formula for Fourier coefficients of cusp forms or of Eisenstein series). 

The above identity is the identity between functionals on V. It is easy to translate it into the identities 
for weighted sums of coefficients 7's and S's. Let v E V be a vector. It will play the role of a test 
function. As we explained in Section 11.1.31 in order to construct model periods, we have to consider 
model realizations of all corresponding representations. In particular, we can view v E V as a function on 
some manifold (or a section of a vector bundle) . The resulting functionals I™* and could be viewed 
as integral transforms on the spaces of such functions. Hence, we obtain for any v E V, the identity 

E w< • r z d M = E • i?°?(v) (1.5) 



(i 
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for the weighted sums of products of automorphic periods 7^,^ = a 7T(Ji ■ ^,,c, and S„ tTj — c W Tj • d Tji c- 
The main point of (11. 5p is that the weights I™%f(v) and I™° d (v) could be computed in some explicit 
models without any reference to the automorphic picture. We will show below that as a special case, 
these identities include the classical Rankin-Selberg identity. 

Remark 1.2. We note that one can associate a non-trivial spectral identity of a kind we described above 
to a pair of different filtrations of a group Q by subgroups forming strong Gclfand pairs. Namely, we can 
associate a spectral identity to two filtrations {T = G C G\ C ■ • • C G n = Q} and {T = H C Hi C 
• ■ ■ C H m = Q} of subgroups in the same group Q such that all pairs (G^+i, Gj) and HA are strong 

Gelfand pairs having the same intersection T ' . One can also "twist" such an identity by a non-trivial 
character or an automorphic representation (\, U x ) of the group T . In this case the resulting identity is 
not for an ^"-invariant junctional Iy?, but for an automorphic period map in the period space V(V W , U x ). 

1.2.1. Bounds for coefficients. The Rankin-Selberg type formulas (|1.5p can be used in order to obtain 
bounds for the coefficients j njtTi and S n<T . (e.g., Theorems 11.31 and II . 5[) . To this end one has to study 
properties of the integral transforms defined by the functionals I™%f and on V. As mentioned in 

Section 11.1.31 the construction of model functionals involves explicit models of representations in some 
spaces of functions (or sections of vector bundles). The model periods T™£f and T™° d are then given 
as integral operators with explicit kernels and the same is true for the resulting model functionals 
and I™° d - These functionals could be defined for all unitary representations it of Q, a of Hi and r of 
H 2 . Hence we obtain a pair of integral transforms h a = I™° d : V mod -> C(Hi), v i-> h a (v) = I™° d (v) 
(here Hi is the unitary dual of Hi and V mod is an explicit model of the representation V) and for the 
triple (0,7^2,-^) the transform g T (v) — I™° d {v). For the classical Rankin-Selberg identity this pair of 
transforms constitutes the pair of the Fourier and the Mellin transforms on the space of (smooth with 
certain decay at infinity) functions on the line R. The latter is the model for the representation 7r of the 
principal series (see Section [3] for more details). 

For applications, one needs to study analytical properties of these transforms. This is a problem in 
harmonic analysis which has nothing to do with the automorphic picture. We study the corresponding 
transforms, in the particular cases under the consideration, in two technical Lemmas 13.61 and 14.61 where 
some instance of what might be called an "uncertainty principle" for the pair of such transforms is 
established. 

The idea behind the proofs of Theorems 11.31 and 11.51 is quite standard by now (and was learned by 
us from [Go]). It is based on the appropriate Rankin-Selberg type identity and the necessary analytic 
information for the corresponding integral transforms (e.g., Lemmas 13.61 and 14. 6|) . Namely, we construct 
a family of test vectors vt £ V mod parameterizes by the real parameter T > 1 such that when substituted 
into the Rankin-Selberg type identity ()1.5|) it will pick up the (weighted) sum of coefficients 7^^ for i in 
a certain "short" interval around T (i.e., the density h a (vr) is concentrated on Hi around representations 
with the parameter of the representation a close to T). We show then that the integral transform g T (vT) 
of such a vector is a slowly changing function on H2 and estimate its support and the size. This allows us 
to bound the right hand side in (jl.5[) using Cauchy-Schwartz inequality and the mean value (or convexity) 
bound for the coefficients <5 ff , r . (e.g., bounds (|4.9|) . (14.10p ). A simple way to obtain these mean value 
bounds was explained by us in BR3] , 

We note that in order to apply this idea to the identity (|1.5[) one needs to have some kind of a positivity 
which is not always easy to achieve. Namely, in order to bound a single coefficient 7 7r ,cr i we have to know 
that terms 7^,^ • I™° d (v) will not cancel each other in the sum (e.g., all terms are non-negative on one 
side of the identity) . As a result of this constraint there are many identities from which it is not clear how 
to deduce bounds for the corresponding coefficients. In the examples that we consider in this paper we 
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choose representations V = V <£> V of the group Q = G x G with V an irreducible unitary representation 
of some other group G. For such representations the necessary positivity is automatic. 

In this paper we implement the above strategy for G = PGI^R) and two cases: for the unipotent 
subgroup N C G and for a compact subgroup K C G. The first case corresponds to the unipotent Fourier 
coefficients and the formula we obtain is equivalent to the classical Rankin-Selberg formula. The second 
case corresponds to the spherical Fourier coefficients which were introduced by H. Peterson a long time 
ago, but the corresponding formula (see Theorem 11.41) has never appeared in print, to the best of our 
knowledge. 

To relate these cases to the above discussion of Rankin-Selberg type spectral formulas, we set Q = GxG, 
H 2 = AG ^ GxG in both cases under the consideration and Hi = NxN, T = AN NxN GxG for 
the unipotent Fourier coefficients and Hi = K x K, T = AK <^-> Kx K °-> GxG for the spherical Fourier 
coefficients. Strictly speaking, the uniqueness principle is only "almost" satisfied for the subgroup N, but 
the theory of the constant term of the Eisenstein series provides the necessary remedy in the automorphic 
setting (see Section . 

We also illustrate analytic applications of these Rankin-Selberg type spectral identities. We prove non- 
trivial bounds for both types of these Fourier coefficients. While bounds for the unipotent coefficients 
(Theorem ll.3|) are known (and even much better bounds are known for the Hecke-Maass forms), for the 
spherical case our bounds (Theorem II. 5[) are new. As a corollary, we obtain a subconvexity bound for 
certain automorphic L-functions. 

The method described above also lies behind the subconvexity for the triple L- function given in p3R4 . 
There the corresponding strong Gelfand formation consists oiQ^GxGxGxG with G = PGL2 (K) , 
T — AG and Hi — G x G with two different embedding into Q. Recently it became clear that there are 
many strong Gelfand formations in higher rank groups. We hope to discuss the corresponding identities 
elsewhere. 



1.3. Unipotent Fourier coefficients of Maass forms. Let G = PGL2(M.) and denote by K — PO(2) 
the standard maximal compact subgroup of G. Let H = G/K be the upper half plane endowed with a 
hyperbolic metric and the corresponding volume element d/xn- 

Let r C G be a non-uniform lattice. We assume for simplicity that, up to equivalence, T has a unique 
cusp which is reduced at 00. This means that the unique up to conjugation unipotent subgroup C F 

is generated by ^ ^ j J ( c -g'-7 T = PSI^Z)). We denote by X = T \ G the automorphic space and by 

Y = X/K = r \ H the corresponding Riemann surface (with possible conic singularities if V has elliptic 
elements). This induces the corresponding Riemannian metric on Y, the volume clement dfiy and the 
Laplacc-Bcltrami operator A. We normalize dfiy to have the total volume one. 

Let 4> T £ L 2 (Y) be a Maass cusp form. In particular, <fi T is an eigenfunction of A with the eigenvalue 

1 _ 2 

which we write in the form /i — — j— for some r£C. We will always assume that (f> T is normalized to 
have L 2 -norm one. We can view <fi T as a V- invariant eigenfunction of the Laplace-Beltrami operator A 
on EL Consider the classical Fourier expansion of <j> T at 00 given by (see |Iwj ) 

T (X + iy) = a n(4>r)Wr,n(y)^ mX ■ (1-6) 

n=iO 

Here W T ,n{y)e 27rmx are properly normalized eigenfunctions of A on EI with the same eigenvalue [i as 
that of the function <j> T , The functions W T .„ are usually described in terms of the JC-Bessel function. In 
Section [3.11 we recall the well-known description of functions W r .„ in terms of certain matrix coefficients 
for unitary representations of G. 
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We note that from the group-theoretic point of view, the Fourier expansion (|1.6|) is a consequence of the 
decomposition of the function c/> T under the natural action of the group N/T^ (commuting with A) . Here 
N is the standard upper-triangular subgroup and the decomposition is with respect to the characters of 
the group N/Too (see Section l3Tj) . 

The vanishing of the zero Fourier coefficient ao(cf) T ) in (|1.6|) distinguishes cuspidal Maass forms (for T 
having several inequivalent cusps, the vanishing of the zero Fourier coefficient is required at each cusp). 

The coefficients a n {4> T ) are called the Fourier coefficients of the Maass form cj> T and play a prominent 
role in analytic number theory. 

One of the central problems in the analytic theory of automorphic functions is the following 

Problem: Find the best possible constants a, p and Cr such that the following bound holds 

M-MI <c r >r -(i + M) p ■ 

In particular, one asks for constants a and p which are independent of <p T (i.e., depend on T only; for a 
brief discussion of the question history, see Remark ll.5.4p . 

It is easy to obtain a polynomial bound for coefficients a n {<j) T ) using boundedness of <j) T on Y. Namely, 
G. Hardy and E. Hecke essentially proved that the following bound 

KOMI 2 <C-max{T,l + |r|} 

|n|<T 

holds for any T > 1, with the constant C depending on T only (see [Iw]). It would be very interesting to 
improve this bound for coefficients a n ((j) T ) in the range \n\ <C |t|. 

For a fixed r, we have the bound \a n ((f> T )\ < C T \n\? . This bound is usually called the standard bound 
or the Hardy/Hecke bound for the Fourier coefficients of cusp forms (in the n aspect). 

The first improvements of the standard bound are due to H. Salie and A. Walfisz using exponential 
sums. Rankin |Ra| and Selberg [Se] independently discovered the so-called Rankin-Selberg unfolding 
method (i.e., the formula (|1.9p below) which allowed them to show that for any e > 0, the bound 
|a„(</>)| <C |tt,| to + e holds. Their approach is based on the integral representation for the weighted sum of 
Fourier coefficients a n ((f>). To state it, we assume, for simplicity, that the so-called residual spectrum is 
trivial (i.e., the Eisenstein series E(s,z) are holomorphic for s S (0, 1); e.g, T = PGL^^f). (The reader 
also should keep in mind that we use the normalization vol(F) = 1 and vol(r<x> \N) = 1.) We have then 

VM^)| 2 a(n) = a(0) + -^ f D{s,<f>,4>)M{a){s)ds , (1.7) 

where a e C°° (R) is an appropriate test function with the Fourier transform a and the Mellin transform 
M(a)(s) , 

D(s, 0, j) = T(s, t) ■ (#, E(s)) L 2 (Y) , (1.8) 

where E(z, s) is an appropriate non-holomorphic Eisenstein series and T(s, t) is given explicitly in terms 
of the Euler T-function (see Remark ll. 5. 4|l . 

The proof of (II. 7|) . given by Rankin and Selberg, is based on the so-called unfolding trick, which 
amounts to the following. Let E(s,z) be the Eisenstein series given by E(s,z) = X) 7 er \ry s (l z ) f° r 
Re(s) > 1 (and analytically continued to a meromorphic function for all s 6 C). We have the following 
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"unfolding" identity valid for Re(s) > 1, 

,E(z, s)) L 2 {Y} = / (j){z)4>{z) ^2 V S il z ) d ^Y = 

J'T (1-9) 
4>{z)4>{z)y s {z)d m = ( <f>(x + iy)4>(x + iy) dx ) y s_1 cPy . 

Too\H JO \JQ J 

The Mellin inversion formula, together with the Fourier expansion (|1.6|l for leads to the Rankin-Selberg 
formula 

Using the strategy formulated in Section [TT2l in this paper we deduce the Rankin-Selberg formula (|1 . 7[1 
directly from the uniqueness principle in representation theory and hence avoid the use of the unfolding 
trick (|1.9j) . One of the uniqueness results we are going to use is related to the unipotent subgroup 
N C G such that C N (the so-called T-cuspidal unipotent subgroup). In fact, the definition of 
classical Fourier coefficients a n {4> T ) is implicitly based on the uniqueness of iV-equivariant functionals 
on an irreducible (admissible) representation of G (i.e., on the uniqueness of the so-called Whittaker 
functional). For this reason, we call the coefficients a n {4> T ) the unipotent Fourier coefficients. 

We obtain a somewhat different (a slightly more "geometric" ) form of the Rankin-Selberg identity (|1.7|) . 
In particular, we exhibit a connection between analytic properties of the function D(s, 4>, 4>) and analytic 
properties of certain invariant functionals on irreducible unitary representations of G. This allows us to 
deduce subconvexity bounds for Fourier coefficients of Maass forms for a general T in a more transparent 
way (here we relay on ideas of A. Good |Go| and on our earlier results [BR1] and [BR3 ]). Namely, we 
prove the following bound for the Fourier coefficients a n (4> T ). 

Theorem 1.3. Let 4> T be a fixed Maass form of L 2 -norm one. For any e > 0, there exists an explicit 
constant C £ such that 

K(<MI 2 <c e -r§+^ . 

\k-T\<ri 

In particular, we have \a„((j) T )\ <C |?i|3+ £ . This is weaker than the Rankin-Selberg bound, but holds 
for general lattices T (i.e., not necessary a congruence subgroup). The bound in the theorem was first 
claimed in |BR1| and the analogous bound for holomorphic cusp forms was proved by Good |Go| by a 
different method. Here we give full details of the proof following a slightly different argument. 

The main goal of this paper, however, is different. Our main new results deal with another type of 
Fourier coefficients associated with a Maass form. These Fourier coefficients, which we call spherical, are 
associated to a compact subgroup of G. 

1.4. Spherical Fourier coefficients. When dealing with spherical Fourier coefficients we assume, for 
simplicity, that T C G is a co-compact subgroup and Y = T \ H is the corresponding compact Riemann 
surface. Let <p T be a norm one eigenfunction of the Laplace-Beltrami operator on Y , i.e., a Maass form. 
We would like to consider a kind of a Taylor series expansion for <fi T at a point on Y. To define this 
expansion, we view (j) T as a T-invariant eigenfunction on EL We fix a point zq £ H. Let z — (r,9), r £ K + 
and 6 G S 1 , be the geodesic polar coordinates centered at Zq (see [He] ) . We have the following spherical 
Fourier expansion of <p T associated to the point zq 

<\> T {z) = Kzo(<l>r)Pr,n(r)e me ■ (1.10) 

Here functions P T ,n(r)e are properly normalized eigenfunctions of A on I with the same eigenvalue 
H as that of the function <f> T . The functions P Ti „ can be described in terms of the classical Gauss 
hypergeometric function or the Legendre function. In Section 14.2.11 we will describe special functions 
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P T n and their normalization in terms of certain matrix coefficients of irreducible unitary representations 
of G. 

We call the coefficients b n (<fr T ) — b n , Zo (4> T ) the spherical (or anisotropic) Fourier coefficients of <j> T 
(associated to a point z n ). These coefficients were introduced by H. Petersson and played a major role 
in recent works of Sarnak (e.g., |SaJ). ft was discovered by J.-L. Waldspurger |Wa| that in certain cases 
these coefficients are related to special values of L- functions (see Remark l"! .5.1 j) . 

As in the case of the unipotent expansion (|1.6|) . the spherical expansion (| 1 . 10[) is the result of an 
expansion with respect to a group action. Namely, the expansion ()1.10[) is with respect to characters of 
the compact subgroup K Zo — Stab Zo G induced by the natural action of G on H (for more details, see 
Section [p. 

The expansion (|1.10|) exists for any eigenfunction of A on H . This follows from a simple separation of 
variables argument applied to the operator A on H. For a proof and a discussion of the growth properties 
of coefficients b n ((j)) for a general eigenfunction <fi on H, see [Hej . [L] , For another approach which is 
applicable to Maass forms, see |BR2j . 

Under the normalization we choose, the coefficients b n (<p T ) are bounded on the average. Namely, one 
can show that the following bound holds 

]T |&„(0 r )| 2 <C'-max{T,l + |r|} 

\n\<T 

for any T > 1, with the constant C depending on T only (see [R]). 

As our approach is based directly on the uniqueness principle, we are able to prove an analog of the 
Rankin-Selberg formula (|1.7p with the group N replaced by a maximal compact subgroup of G. This is 
the main aim of the paper. We obtain an analog of the Rankin-Selberg formula (|1.7jl for the coefficients 
bni^r)- Roughly speaking, new formula amounts to the following (for the exact form, see formula lj4.8|l ) 

Theorem 1.4. Let {0a, } be an orthonormal basis of L 2 (Y) consisting of Maass forms. Let <f> T be a fixed 
Maass form. 

There exists an explicit integral transform * : C°°(S 1 ) — > C°°(C), u(9) i— > it*. (A), such that for all 
u 6 C°°(S ), the following relation holds: 

\b n {4>r)\ 2 u(n) = u (l) + J2 ^o(^) • «»(Ai) , (1.11) 
n A«#l 

with some explicit coefficients C Zo ((f>Xi) £ C which are independent of u. 

Here u(n) = J u(9)e~ m9 d9 and u(l) is the value at 1 6 S . 
77 s 1 

The definition of the integral transform " is based on the uniqueness of certain invariant trilinear 
functionals on irreducible unitary representations of G and described explicitly in the formula (|4.7ll . The 
trilinear functional was studied by us in BR3] and |BR4j . The main point of the relation (|1.11|) is that 
the transform uf.(Aj) depends only on the parameters \i and r, but not on the choice of Maass forms <p\ i 
and <j) T . The coefficients C Zo (4>\ i ) are essentially given by the product of the triple product coefficients 
0A i )_L 2 (i') an d the values of Maass forms 4*^ at the point zq. In some special cases both types of 
these coefficients are related to L-functions (see [W], jJNj . [Waj and Remark ll.5.ip . 

A formula similar to (jl.lljl holds for a non-uniform lattice T as well, and includes the contribution 
from the Eisenstein series (see the formula in Remark l4.5p . Also, a similar formula holds for holomorphic 
forms. We intend to discuss it elsewhere. 
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The new formula allows us to deduce the following bound for the spherical Fourier coefficients 

of Maass forms. 

Theorem 1.5. Let T be as above and <p T is a fixed Maass form of L 2 -norm one. For any e > 0, there 
exists an explicit constant D £ such that 

E M<MI 2 <ivtI+- . 

fc-T|<T§ 

In particular, we have \b n (<p T )\ -C |n|^ +£ for any e > 0. An analogous bound should hold for spherical 
Fourier coefficients of holomorphic cusp forms. We hope to return to this subject elsewhere. 

The proof of the bound in the theorem follows from essentially the same argument as in the case of the 
unipotent Fourier coefficients, once we have the Rankin-Selberg type identity In the proof we use 

bounds for triple products of Maass forms obtained in |BR3j , and a well-known bound for the averaged 
value of eigenfunctions of A. 

In special cases, the bound in the theorem could be interpreted as a subconvexity bound for some 
automorphic L- function (see Remark ll.5.1j) . 



1.5. Remarks. 



1.5.1. Special values of L- functions. One of the reasons one might be interested in bounds for coefficients 
bk{<t>r) is their relation to certain automorphic L-functions. It was discovered by J.-L. Waldspurger |Waj 
that, in certain cases, the coefficients &fc(0 r ) are related to special values of L-functions. H. Jacquet 
constructed the appropriate relative trace formula which covers these cases (see |JN| ). The simplest case 
of the formula of Waldspurger is the following. Let zo = i G SL2 (Z) \ H and E = Q(i). Let tt be the 
automorphic representation which corresponds to 4> T , II its base change over E and Xn{z) — (z/z) An 
the n-th power of the basic Grossencharacter of E. One has then, under appropriate normalization (for 
details, see |Waj . |JN] . |MWj ). the following beautiful formula 

1 nWMI L(l,Adir) ■ [ ' 



Using this formula, we can interpret the bound in Theorem 11.51 as a bound on the corresponding L- 
functions. In particular, we have |L(i,n®Xn)l < |«| 2/3+e . This gives a subconvexity bound (with 
the convexity bound for this L-function being |L(i,II (g) Xn)\ *C |n| 1+e ). The exponent in the bound 
corresponds to what is known as a H. Weyl type subconvexity bound for an L-function. 

The subconvexity problem is a classical question in analytic theory of L-functions which received much 
of attention in recent years (we refer to the survey [IS] for the discussion of subconvexity for automorphic 
L-functions). In fact, Y. Petridis and P. Sarnak |PS| recently considered more general L-functions. 
Among other things, they have shown that |L(| + ito,H (8> Xn)\ \n\ ws +e for any fixed to € K. and 
any automorphic cuspidal representation II of GL2(E) (not necessarily a base change). Their method 
is also spectral in nature although it uses Poincare series and treats L-functions through (unipotent) 
Fourier coefficients of cusp forms. We deal directly with periods and the special value of L-functions only 
appear through the Waldspurger formula. Of course, our interest in Theorem 1 1 . 51 lies not so much in the 
slight improvement of the Petridis-Sarnak bound for these L-functions, but in the fact that we can give 
a general bound valid for any point zq. (It is clear that for a generic point or a cusp form which is not a 
Hecke form, coefficients b n are not related to special values of L-functions.) 

Recently, A. Venkatesh [V] announced (among other remarkable results) a slightly weaker subconvexity 
bound for coefficients b n (<j) T ) for a fixed <p T . His method seems to be quite different and is based on 
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ergodic theory. In particular, it is not clear how to deduce the identity p. lip from his considerations. On 
the other hand, the ergodic method gives a bound for Fourier coefficients for higher rank groups (e.g., on 
GL(n)) while it is not yet clear in what higher-rank cases one can develop Rankin-Selberg type formulas 
similar to (jl.ll[) which would lead to bounds for the corresponding coefficients. 

1.5.2. Fourier expansions along closed geodesies. There is one more case where we can apply the unique- 
ness principle to a subgroup of PGL2 (K) . Namely, we can consider closed orbits of the diagonal subgroup 
A C PGL2{M.) acting on X. It is well-known that such an orbit corresponds to a closed geodesic on Y (or 
to a geodesic ray starting and ending at cusps of Y) . Such closed geodesies give rise to Rankin-Selberg 
type formulas similar to ones we considered for closed orbits of subgroups N and K. In special cases the 
corresponding Fourier coefficients are related to special values of various L- functions (e.g., the standard 
Hecke i-function of a Hecke-Maass forms which appears for a geodesic connecting cusps of a congruence 
subgroup of PSL(2, Z)). In fact, in the language of representations of adele groups, which is appropriate 
for arithmetic T, the case of closed geodesies corresponds to real quadratic extensions of Q (e.g., twisted 
periods along Heegner cycles) while the anisotropic expansions (at Heegner points) which we considered 
in Section 11.41 correspond to imaginary quadratic extensions of Q (e.g., twisted "periods" at Heegner 
points). 

In order to prove an analog of Theorems 11.31 and 11.51 for the Fourier coefficients associated to a closed 
geodesic, one has to face certain technical complications. Namely, for orbits of the diagonal subgroup A 
one has to consider contributions from representations of discrete series, while for subgroups N and K 
this contribution vanishes. It is more cumbersome to compute a contribution from discrete series as these 
representations do not have nice geometric models. Hence, while the proof of an analog of Theorem ll.4l 
for closed geodesies is straightforward, one has to study invariant trilinear functionals on discrete series 
representations more closely in order to deduce bounds for the corresponding coefficients. We hope to 
return to this subject elsewhere. 

1.5.3. Dependence on the eigenvalue. From the proof that we present it follows that the constants C £ 
and D £ in Theorems 11.41 and 11.51 satisfy the following bound 

C e , £> £ <C(r)-(l + |r|)-|lne| , 

for any < e < 0.1, and some explicit constant C(T) depending on the lattice T only. 

1.5.4. Historical remarks. The question of the size of Fourier coefficients of cusp forms was posed (in the 
n aspect) by S. Ramanujan for holomorphic forms (i.e., the celebrated Ramanujan conjecture established 
in full generality by P. Deligne for the holomorphic Hecke cusp form for congruence subgroups) and was 
extended by H. Petersson to include Maass forms (i.e., the Ramanujan-Petersson conjecture for Maass 
forms). In recent years the r aspect of this problem also turned out to be important. 

Under the normalization that we have chosen, it is expected that the coefficients a n {4> T ) are at most 
slowly growing as n — > 00 ( jSaj ). Moreover, it is quite possible that the strong uniform bound of the form 
|«Tt(0r)| <C (M(l + \t\)) 6 holds for any e > (e.g., Ramanujan-Petersson conjecture for Hecke-Maass 
forms for congruence subgroups of PSL2CZ)). We note, however, that the behavior of Maass forms and 
holomorphic forms in these questions might be quite different (e.g., high multiplicities of holomorphic 
forms). 

Using the integral representation (jTTTJ) and detailed information about Eisenstein series available only 
for congruence subgroups, Rankin and Selberg showed that for a cusp form cj) for a congruence subgroup 
of PGL 2 {%) one has X]| n |< T |a„(0)| 2 = CT + 0(T 3 / 5+e ) for any e > 0. In particular, this implies that 

for any e > 0, |a n (0)| <C |n|^ +e . Since their groundbreaking papers, this bound was improved many 
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times by various methods (with the current record for Hecke-Maass forms being 7/64 » 0.109... due to 
H. Kim, F. Shahidi and P. Sarnak [KSa] L 

The approach of Rankin and Selberg is based on the integral representation of the Dirichlet series given 
for Re(s) > 1, by the series D(s, 0, 4>) = J2 n >o '""n^' ■ 



The introduction of the so-called Ranking-Selberg L-function L(s, (j> ® (f>) — ((2s)D(s, <fi, <fi) played an 
even more important role in the further development of automorphic forms than the bound for Fourier 
coefficients which Rankin and Selberg obtained. 

Using integral representation (|1.8[) . Rankin and Selberg analytically continued the function L(s, <j>® 4>) 
to the whole complex plane and obtained an effective bound for the function L(s, <j> <E> (f>) on the critical 
line s = h + it for T being a congruence subgroup of 5*L 2 (Z). From this, using standard methods in 
the theory of Dirichlet series, they were able to deduce bounds for Fourier coefficients of cusp forms. 
In fact, Rankin and Selberg appealed to the classical Perron formula (in the form given by E. Landau) 
which relates analytic behavior of a Dirichlet series with non-negative coefficients to partial sums of its 
coefficients. The necessary analytic properties of L(s, <t>®<t>) are inferred from properties of the Eisenstein 
series through the formula (JT7 



A small drawback of the original Rankin-Selberg argument is that their method is applicable to Maass 
(or holomorphic) forms coming from congruence subgroups only. The reason for such a restriction is the 
absence of methods which would allow one to estimate unitary Eisenstein series for general lattices T. 
Namely, in order to effectively use the Rankin-Selberg formula (|1.7[) one would have to obtain polynomial 
bounds for the normalized inner product L>(s,0, <j)) = T{s,t) ■ (<f>4>, E{s)) L i(y)- This turns out to be 
notoriously difficult because of the exponential growth of the factor T(s. r) = r 2 (s/2)r(s/2+T/2)r(s/2-T/2) > 
for \s\ — ► oo, s G iM.. For a congruence subgroup, the question could be reduced to known bounds for the 
Riemann zeta function or for Dirichlet L-functions, as was shown by Rankin and Selberg. The problem 
of how to treat general T was posed by Selberg in his celebrated paper [Sej . 

The breakthrough in this direction was achieved in works of Good |Goj (for holomorphic forms) and 
Sarnak [Sa] (in general) who proved non-trivial bounds for Fourier coefficients of cusp forms for a general 
r using spectral methods. The method of Sarnak was finessed in [BR1] by introducing various ideas from 
the representation theory and further extended in KS] . The method of this paper is different and avoids 
the use of analytic continuation which is central for |Sa| . [BRlj and [KSj . 

Special cases of the Rankin-Selberg spectral identities described in the introduction were obtained 
before by a different method. The first (vague) attempt to write the above-mentioned formula for four 
copies of G = PGL2(M.) and all representations coming from Eisenstein series was made by N. Kuznetsov 
[Kz] , His aim was to obtain a formula representing the eighth moment of the Riemann zeta function. 
Later Y. Motohashi [Molj obtained the formula for the fourth moment of the Riemann zeta function. 
This corresponds to our identity with Q = G x G for G = PGL 2 (M), H x = T x T where T C G is the 
diagonal subgroup, T = AT and H2 = AG. To obtain the fourth moment of the Riemann zeta function, 
Motohashi considers representations coming from Eisenstein series. This leads to considerable technical 
difficulties which one should not underestimate. Both Kuznetsov and Motohashi based their approach 
on the celebrated Bruggeman-Kuznetsov trace formula (applying it twice!). The setup we present here 
even if it does not simplify the arguments at least gives a more conceptual explanation for the terms 
appearing in these identities. 

Many other cases of these identities appeared more recently (mostly stated implicitly as a tool for 
estimation of L-functions or other quantities). Among these are works of R. Bruggeman, V. Bykovskii, 
A. Ivic, M. Jutila, P. Michel, A. I. Vinogradov to name a few. 
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Finally, we would like to mention that recently R. Bruggeman, M. Jutila and Y. Motohashi (see jBMj , 
|Mo2| and references therein) developed what they call the inner product method. It is based on the 
unfolding of an appropriate Poincare or Petersson type series. The standard unfolding leads to the 
spectral expansion for the series of the type J^k Ak(<t>)Ak+h{(f>)W(k), where is a Maass form and 
are appropriate Fourier coefficients (e.g., unipotent or spherical Fourier coefficients we discussed above). 
The formulas obtained in such a way are special cases of our Rankin-Selberg type formula (|3.6p for a 
special test vectors v. These vectors are constructed from certain functions on the upper-half plane. As 
a result, the corresponding weights in the Rankin-Selberg type formulas are reminiscent of exponential 
weights considered by Selberg and Rankin. It seems that using our approach one can avoid a difficult 
task of removing these unwanted weights. 

The paper is organized as follows. In Section [21 we quickly recall the notion of automorphic represen- 
tations of G and describe the standard models of representations we will use. 

In Section [3] we reprove the classical Rankin-Selberg formula and deduce bounds for the unipotent 
Fourier coefficients of Maass forms. The proof is based on the uniqueness of trilinear invariant functionals 
on irreducible unitary representations of G. We use the description of these functionals obtained in | BR3 . 

In Section |4] we apply the same strategy to the spherical Fourier coefficients. In fact, in this case the 
proof is less involved since we do not need the theory of the Eisenstein series in order to remedy the 
non-uniqueness of A-invariant functionals on irreducible representations of G. Section 3] contains our 
main new results and the reader might read this section independently of Section [3l 

In the appendix we prove an asymptotic expansion of the model trilinear functional. We use this 
analysis in the proof of Theorem 11.51 

Acknowledgments. This paper is a byproduct of a joint work with Joseph Bernstein. It was written 
under his insistence. It is a special pleasure to thank him for numerous discussions and for his constant 
encouragement and support over many years. I also would like to thank Peter Sarnak for stimulating 
discussions and support. 

2. Representations of PGL 2 (R) 

We start with a short reminder about the connection between Maass forms and representation theory 
of PGL 2 (R) which is due to Gelfand and Fomin (see |G6| ). 

2.1. Models of representations. All irreducible unitary representations of the group G = PGA 2 (R) 
are classified. For simplicity we consider those with a nonzero AT-fixed vector (so-called representations of 
class one) since only these representations arise from Maass forms. These are the representations of the 
principal and the complementary series and the trivial representation. We will use the following standard 
model (or realization) for these representations. 

For every complex number r, consider the space V T of smooth even homogeneous functions on R 2 \ 
of the homogeneous degree r — 1 (which means that f(ax,ay) = \a\ T ~ 1 f(x,y) for all a S R\ 0). The 
representation (jr T , V T ) is induced by the action of the group GA 2 (R) given by the formula n T (g)f(x, y) = 
f{g^ 1 (x, y))\ det g\~^~ ■ This action is trivial on the center of GA 2 (R) and hence defines a representation 
of G. The representation (jr T , V T ) is called representation of the generalized principal series. 

For explicit computations it is often convenient to pass from the plane model to a line model. Namely, 
the restriction of functions in V T to the line (x, 1) C R 2 defines an isomorphism of the space V T with 
the space G£°(R) of restrictions of smooth homogeneous functions (e.g., decaying at infinity as |a;| T_1 ). 
Hence we can think about vectors in V T as functions on R. 
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In the line model the action of an element a = diag(a, a 1 ), a G R x , in the diagonal subgroup is given 



unipotent group is given TT T (n)f(x, 1) = f(x — n, 1) . 

When r = it is purely imaginary, the representation (t: t ,V t ) is pre- unitary and irreducible; the G- 
invariant scalar product in V T is given by (/, g)y T = \ J R fgdx. These representations are called the 
principal series representations. 

When t G (—1, 1), the representation (jt Tl V T ) is called a representation of the complementary series. 
These representations are also pre-unitary and irreducible, but the formula for the scalar product is more 
complicated (see |G5j V 

All these representations have if-invariant vectors. We fix a A"- invariant unit vector e T G V T to be 
a function which is constant on the unit circle S 1 in R 2 in the plane realization. Note that in the line 
model a AT-fixed unit vector is given by e T (x) = c(l + x 2 )^~ with |c| 2 = n^ 1 for r € iR. 

Another realization, which we call circle or spherical model, is obtained by restricting functions in V T to 
the unit circle S* 1 C R 2 \ 0. In the circle model we have the isomorphism V T ~ C^ en (S 1 ) and for r G iM., 
the scalar product is given by (/, g) = ^ J sl fgdO while the action of K is induced by the rotation of S 1 . 

Representations of the principal and the complimentary series exhaust all nontrivial irreducible pre- 
unitary representations of G of class one. 

2.2. Automorphic representations. Every automorphic form cf> generates (under the right translations 
by elements in G) an automorphic representation of the group G (see [G6J; this means that, starting 
from (/), we produce a smooth irreducible unitarizable representation of the group G in a space V and 
its realization v : V — > C°°(X) in the space of smooth functions on the automorphic space X = T\G. 

We will denote by V T the isomorphism class of the representation arising in this way from a Maass form 

i — 2 

4> = 4> T with the eigenvalue fi = 4 T . 

Suppose we are given a class one unitary representation (t, V t ) and an automorphic realization of it 
v : V T — > C°°(X); we assume v to be an isometric embedding. Such v gives rise to an eigenfunction of 
the Laplacian on the Riemann surface Y = X/K as before. Namely, if e T G V T is a unit K-fixed vector, 
then the function <f> = v(e T ) is a L 2 -normalized eigenfunction of the Laplacian on the space Y = X/K 
with the eigenvalue /i = 4 r . This explains why r is a natural parameter to describe Maass forms. 

This construction gives a one-to-one correspondence between Maass forms and class one automorphic 
representations (and more generally between automorphic forms and automorphic representations of G) . 
We use this correspondence to translate problems in automorphic forms into problems in representation 
theory. 



3.1. Whittaker functionals. We start with the classical interpretation of Fourier coefficients afc(^ T ) 
in terms of representation theory. Namely, we consider Whittaker functionals on irreducible unitary 
representations of G. 

Let 4> be a Maass form and v : V — > C°°(X) the corresponding automorphic realization of the space of 
smooth vectors of an irreducible unitary representation of G. 

Let N C G be the standard upper-triangular unipotent subgroup. We denote by M the A^-invariant 
closed cycle (a horocycle) \ N C X, The cycle Af is an orbit Af = e ■ N C X of N, where e is the 



by 7r T (a) f(x, 1) = /(a 1 x, a) — \a\ T 1 f(a 2 x, 1) ; and the action of an element h 
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image of the identity element e G G under the natural projection G — > X. In what follows we can choose 
any closed orbit of any unipotent subgroup of G. We endow AT with the TV-invariant measure dn of the 
total mass one, and fix an identification r m \ N ~ Z \ R. 

For fc G Z, let ^fc : AT -> C be the additive character tp k (t) = e 27Ttkt of N ~ R trivial on ~ Z C K . 
We consider the functional 1% — 1°^ : V — > C defined by the automorphic period 

Zfe(u) = / ^(u)(ri)-0fe(n)dn for any n£l/. 

The functional If, G is (JV, -0fc)-equi variant, i.e., ^(7r(n)u) = ipk(n)lf,(v) for any n £ N and v £ V. 
It is well-known that for a non-trivial character the space of functionals in V* satisfying this property 
is one-dimensional. The automorphic representation (V, v) is called cuspidal if lfj jQ = (for any cuspidal 
subgroup Tjv). We also have the standard Fourier expansion of cuspidal automorphic functions along TV: 

«/(»)(*) = £i*Ws)»). 

where g corresponds to x under the projection p:G^>T\G = X (i.e., p(g) = x). 

We now consider model Whittaker functionals. In the line model of the representation V = V T C C°° (R) , 
we can construct a model Whittaker functional I™ = l™° d : V — ► C by using the Fourier transform. 
Namely, let v G V C C°°(R) be a vector (i.e., a smooth function) of a compact support and £ G R. We 
define the model Whittaker functional by the integral 

l™(v) = = [ v{x)e-* x dx . 
Js. 

The functional I™ clearly extends to the whole space V by continuity. 

The uniqueness of Whittaker functionals implies that the model and the automorphic functionals are 
proportional. Namely, for any k G Z \ 0, there exists a constant dk{v) G C such that 

l% = a k {y)-lV . 

A simple computation shows that under our normalization |afc(^)| = |afc(0 T )|. Namely, we have 
Z™(e r ) = f(l + t 2 ) 2 ^ exp(— it;t)dt = ^Jj^L 7 K_ r / 2 (£), where K t denotes the i^-Bessel function. 
Based on this we choose in (|1.6j) the following normalization for the Whittaker functions 

WrAv) = I'Z" ("* y -i ) = r^fj ' V~ h K_ T/2 (2n\k\y) . 

Under such normalization we have <Xfc(</>) = ak{v), and this is consistent with one of the classical normal- 
izations for Fourier coefficients of Maass forms (see [Iwj ) . 

3.2. Weighted sums of coefficients. We are interested in bounds for Fourier coefficients atk{v)- To 
this end we consider bounds for weighted sums of the type 

^2 \a k {v)\ 2 6t{k), 

k 

where a is a non-negative weight function. There is a simple geometric way to construct these sums. 

Let V be the representation which is complex conjugate to V; it is also an automorphic representation 
with the realization v : V — > C°°(X). We only consider the case of representations of the principal series, 
i.e., we assume that V = V T , V = V— T for some r G iR; the case of representations of the complementary 
series is similar. 
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Consider the space E = V ® V. We identify it with a subspace of C°°(R 2 ) using the line realization 
V C G°°(R). We have the corresponding automorphic realization v ^ = v ®v : E = V ' ®V — > C°°(X xX). 

Let AM C AX C X x X be the diagonal copy of the cycle TV. We define the following automorphic 
iV-invariant functional Zaa/" : E — > C by 

Iaat(w) = / v E (w)(n,n)dn 
J AM 

for any w E E. 
We have the obvious Plancherel formula 

fc (3.1) 

k k 

for any w E E C G°°(R 2 ) (here w denotes the standard Fourier transform on R 2 ). 

Varying the vector w € E, we obtain different weighted sums |afe(^)| 2 d(fc) with a weight function 
a(h) = w(k, —k). The weight function might be easily arranged to be non-negative as we will see below. 

We now obtain another expression for the functional ^ajV using spectral decomposition of L 2 (X) and 
trilincar invariant functionals on irreducible representations of G. This will give an instance of the Rankin- 
Selberg type formula discussed in Introduction which in fact coincides with the classical formula of Rankin 
and Selberg. We first discuss spectral decomposition of L 2 (X) into irreducible unitary representations of 
G. 



3.3. Spectral decomposition and the Eisenstein series. It is well-known that L 2 (X) = L 2 usp (X) © 
L 2 es (X) © L 2 Eis {X) decomposes into the sum of three closed G-invariant subspaces of cuspidal repre- 
sentations, representations associated to residues of Eisenstein series and the space generated by the 
unitary Eisenstein series (see [B]). The spaces L 2 usp {X) and L 2 es (X) decompose discretely into a direct 
sum of irreducible unitary representations of G and L Eis (X) is a direct integral of irreducible unitary 
representations of the principal series. We assume for simplicity that the residual spectrum is trivial, i.e., 
L 2 es (X) — C is the trivial representation of G (e.g., V is a congruence subgroup of PSL^i^))- 

We are interested in the spectral decomposition of the functional Zaa/" defined as a period along the 
diagonal copy of a horocycle inside of X x X. Hence, the space L 2 usp (X) will not appear in the final 
formula as by the definition it consists of functions satisfying Jj^f(nx)dn = for almost all x £ X. 

We will need the following basic facts from the theory of the Eisenstein series (see [Be], [B], |Kuj ) . 

Let B = AN be the Borel subgroup of G (i.e., the subgroup of the upper triangular matrices). We 
denote F# = T f~l B, Tn = = T (~l N and F^ = Ts/Ljv which we assume, for simplicity, is trivial. Let 
Aff = N \G ~ {M 2 \ 0}/{±l} be the basic affine space. The group G acts from the right on the space 
Aff and preserves an invariant measure f^Aff- The subgroup B/N acts on Aff on the left and acts on 
HAff by a character. 

We denote Xb = ^bN \ G and endow it with the measure fix B compatible with the measure fix- We 
identify Xb with Aff (in general, one considers the space F/, \ Aff). 

Let A(Xb) be the space of smooth functions of moderate growth on Xb- For a complex number s G C, 
we denote by A s (Xb) — A s (Aff) ~ A s even (R 2 \ 0) the subspace of the even homogeneous functions of the 
homogeneous degree s — 1. The subspace A s (Xb) is G-invariant and for s purely imaginary is isomorphic 
to the space of smooth vectors of a unitary class one representation of G. 
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In this setting one have the Eisenstein series operator 

E : A(X B ) -> C°°(X) 

given by E(/) = J2yer/r B 7° / anc ^ the conjugate constant term operator C : C°°{X) — > .A(Xb) given 
by C(<f>) — J neN / rN nocj) dn. The operator E is only partially defined as the Eisenstein series not always 
convergent. 

The operators E and G commute with the action of G. Hence we also have the operator E(s) = 
E|.a»(Xb) : A s (Xb) — > G°°(X) (defined via the analytic continuation for all s £ iM.) and the fundamental 
relation C(s) o E(s) = Id + I(s) where J(s) : ,A s (Xb) — > A~ S (X B ) is an intertwining operator which 
is unitary for s £ iR. It is customary to write it in the form I(s) — c(s)I s where I s is a properly 
normalized unitary intertwining operator satisfying I s o I_ s = Id and c(s) is a meromorphic function, 
satisfying the functional equation c{s)c(—s) = 1. The operator J s is constructed explicitly in a model of 
the representation V s . We also have the functional equation E(s) = E(— s) o/(s) for the Eisenstein series. 

The spectral decomposition of L 2 Eig (X) then reads 

L 2 Eis (X) = [ E(s)(A s (X B )) ds . 

This means, in particular, that for any / £ C°°{X) n L 2 (X), the Eisenstein component Jeis = P r Eis(f) 
of / in the space L Eis (X) has the following representation f^ s = J iR+ E(s)/ S (is for an appropriate 
smooth family of functions / s £ A s (Xg). We choose an orthonormal basis {e,(s)} C ^l s (Xs) and set 
/ s = E(s)ej(s))i2(jf)ej(s) for all s € iM. We have then a more symmetrical spectral decomposition 

f Eis = \ [ E(s)f s ds , 
1 Jim. 

and the corresponding Plancherel formula | |/sis | lz, 2 (^r) = 5 JiM H/ s ll!/t s (x B ) 



3.4. Trilinear invariant functionals. We construct the spectral decomposition of l&Af with the help 
of trilinear invariant functionals on irreducible unitary representations of G. We review the construction 
below (for a more detailed discussion see |BR3] V 

Let v : V — > C°°(X) be a cuspidal automorphic representation. Let E = V (E> V and ve be as above. 
Consider the space C°°(X x X). The diagonal AX X x X gives rise to the restriction morphism 
r\ : C°°(X x X) — > C°°(X). Let ^ : — > C°°(X) be an irreducible automorphic subrepresentation. 
We assume that for any w £ W the function Vw{w) is a function of moderate growth on X. We define 
the following G-invariant trilinear functional l E u @ w = K^®v w on E ®W via 

l a E% w { v ® v ' ® u ) = (rA(v®v'),u) L 2 ix) 

for any v ® v' £ E and u £ W . The cuspidality of V and the moderate growth condition on W ensure 
that l E u l w is well-defined (i.e., the integral over the non-compact space X is absolutely convergent). 

Next we use a general result from representation theory, claiming that such a G-invariant trilinear 
functional is unique up to a scalar (see [O], [Pr] and the discussion in |BR3j ). Namely, we have the 
following 

Theorem. Let (irj,Vj), where j = 1,2,3, be three irreducible smooth admissible representations of G. 
Then dimHom G (Vi (8) V 2 <8> V 3 , C) < 1. 

This implies that the automorphic functional l%%w is proportional to an explicit "model" functional 
Ie®W wmcn we describe using explicit realizations of representations V and W of the group G; it is 
important that this model functional carries no arithmetic information. The model functional is defined 
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on any three irreducible admissible representations of PGL 2 (M.) regardless whether or not these are 
automorphic . 

Thus we can write 

jaut jmod /o o\ 

for some constant Oe®w — a v E ®v w (somewhat abusing notations as this coefficient depends on the 
realizations i>e and vw and not only on the isomorphism classes of E and W). 

It turns out that the proportionality coefficient aE®w above carries important "automorphic" infor- 
mation (e.g., essentially is equal to the Rankin-Selberg L- function) while the second factor carries no 
arithmetic information and can be evaluated using explicit realizations of representations V and W (see 
Appendix in p3R3 for an example of such a computation) . In what follows we only need the case of W 
being an irreducible unitary representation of the principal series V s ,s£ zR (or the trivial representation) . 

Denote by l™ od the model trilinear form l" lod : V <8> V <8> V s — > C which we describe explicitly in 
Section 13.51 below. Any G-invariant form I : V (£> V ® V s — > C gives rise to a G-intertwining morphism 
T . V ® V — > V* which extends to a G-morphism T l : E —* V s , where we identify the complex conjugate 
space V s with the smooth part of the space V* (V s ~ V- s for s £ iM.). We apply this construction in 
order to describe the projection of E to the space orthogonal to cusp forms, namely to C ffi L Eis (X) (in 
general to L 2 res (X) L% is {X)). 

We realize the irreducible principal series representation V s in the space of homogenous functions 
on the plane „4 S ({K 2 \ 0}/{±l}) ~ A s {Aff) ~ A S (X B ). This is a model suitable for the theory of 
Eisenstein series. For a chosen family of G-invariant functionals Z™ od = Ie®v b '■ E ® V~ s —> C and 
the corresponding family of morphisms T s = T ls : E — > V s o± A s (Xb), we have the proportionality 
coefficient a(s) = a v {s) — aE®V- s defined by l^ ut = a(s) l™ od as in ()3.2() and the corresponding spectral 
decomposition for any w £ E, 

pr res ®Eis{vE{w)) = (r A (i / s(w)), 1) • 1 + - / a(s)E(s)(T s (w)) ds . (3.3) 

1 JiR 

Note that (|3.3|) is symmetrical under the change s — > —s. This is achieved by choosing the model trilinear 
functionals l™ od : E <g> V s -> C to satisfy the relation l™ od = I'^f o I s and the coefficients a(s) to satisfy 
a(s) = c(s)a(—s) (this is equivalent to the functional equation for the Rankin-Selberg L-function). We 
note also that (rA(i>E («>)), 1} = Tr{w) for any w £ E, viewed as an element in V ® V* . 



We use spectral decomposition (J3T3J) to obtain the spectral decomposition of the functional IajV- 

Namely, consider the functional : C°°(X) — ► C given by the constant term along Af C X (i.e., 
'ajv(Z) = C(/)|x=e for any / € C°°(X)). As Ij^f vanishes on L^ usp (X), we only have to understand its 
form on the space of the Eisenstein series (and on the space of residues) . The pair (G, N) is not a Gelfand 
pair (the space of N -invariant functionals is two dimensional) and we can not use the argument we used 
for the Whittaker functionals. However, the theory of the Eisenstein series provides the necessary remedy. 
Namely, consider the representation of the (generalized) principal series A s realized in the space of even 
homogenous functions onlg ~ R 2 \ 0. The space of A- invariant functionals on A s is generated by the 
functionals S s and 5- s , where S s (v) — v(0, 1) and 5- s (v) = l s (v)(0, 1) (in fact, the functional S- s is given 
(up to a normalization constant) by the integral over the line {(1, x)\ x £ R} C K 2 ). The basic theory of 
the constant term of the Eisenstein series then implies that 

C(E(s)(v))\ x=s = 6 s (v) + c(s)6- s (v) . 
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Applying this to (|3.3|) . we obtain the following spectral decomposition 

Iaat(ve(w)) = lM{pr res(S Eis{vE{w))) = V ° l }\[i ■ Tr(w) + [ a(s)S s (T s (w)) ds . 

vol(X) 2 J iR 

Here we have used the functional equation for the constant term of the Eisenstein series 

a(s)c(s) ■ S- s {T s (w)) = a(-s) ■ S- S (T- S (w)) 

and the assumption that the residual spectrum is trivial. Taking into consideration the Plancherel formula 
(|3.ip and the normalization of measures vol(X) = vol(M) — 1, we arrive at the identity 

V \a k {v)\ 2 w(k, -k) = Tr(w) + [ a(s)6 s (T s (w)) ds . (3.4) 
k 

This is our form of the Rankin-Selberg formula. To give it a more familiar form similar to (|1.7[) . we 
will make (|3.4p more explicit by describing T s and S s in the line model of V s . We do this by choosing an 
explicit kernel for the model invariant trilinear functional l" Lod . 



3.5. Model trilinear functionals. We recall the construction of model trilinear functionals presented 
in [BR3] . There it was shown that in the line model of representations V ~ V T , V ~ V- T and V- s the 
kernel 

K r ,-r,-s(x,y,z) = 

\ x _ y\<.->-W\ xz - 1 |(-ar+.-l)/2| |/2 , _ 1 |(2r +S -l)/2 ( 3 ' 5 ) 

defines a nonzero trilinear G- invariant functional Z™ od oe 7 t ® V- T ® V- s . This gives rise to the map 
T s : E T ~ V T ® V-t — > V s given by the same kernel. Here the variable x corresponds to the representation 
V T and y, z to V- T and V- s , respectively. Note a certain asymmetry between V T , V- T and V- s . This is 
because we choose for the first two representations the line model associated with the upper triangular 
subgroup and for the last representation the model associated with the lower triangular subgroup. 

In the line model V s C C°°(M) the N- invariant functional S s is given by the evaluation at the point 
z = 0: S s (f) = /(0). Hence from (|3.5p it follows that the composition 5 S (T S ) is given by the Mcllin 
transform for any w G E C C°° (i x 1), 

S s (T s (w)) = f w(x,y)\x-yt s ^/ 2 dxdy . 

Plugging this into (|3.4[) . we arrive at the "classical" Rankin-Selberg formula (we assume as before that 
the residual spectrum is trivial) 



V M^)| 2 w(£;, -k) = Tr{w) + [ a(s)w\s) ds , 

7 JiR 



(3.6) 



,.b/ 



(a) = / wix^^x-yt^^dxdy . (3.7) 



where we denoted 



This coincides with the Mellin transform M(a)(s) for a(t) — f x __ t w(x, y)dl. The transform b is defined 
for any smooth rapidly decreasing function w, at least for all Re(s) < 1. In fact, it could be defined for 
all A G C, by means of analytic continuation, but we will not need this. We will consider only the case 
s G as we assume that the residual spectrum is trivial. In general, the residual spectrum could be 
treated similarly. We note also that Tr(w) = J w(x,x)dx = a(0). 
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We can now rewrite the Rankin-Selberg formula in a more familiar form if we substitute the vector 
w(x,y) by a(t) = J xy=t w(x,y)dl. We have 

^\a k (v)\ 2 a{k)=a(0)+ [ a(s) ■ M(a)(s) ds , (3.8) 
k 

where a(£) — w(£, —£)■ This coincides with the classical Rankin-Selberg formula. 

3.5.1. Remarks. 1. Taking into account that M(a)(s) = j(s)M(a)(l — s), where 7(s) = ^i-!^ 2 " 1 — 

7r__J_r ( J -r £ ) 

(note that |7(s)| = 1 for s 6 iM.), we see that 

\a k {v)\ 2 a{k) = a(0) + / a(s)<y(s)M(a)(s) ds . (3.9) 

This seems to have an advantage of being an identity for one function a and not for two functions a and 
M(a). In practice we find it easier to work with one master function a and use the identity (|3.8p . 

2. We would like to point out one essential difference between the classical Rankin-Selberg formula 
(|1.7|) obtained via the unfolding and formula (|3.8fl that we prove. The unfolding method provides an 
explicit relation between a choice of a model Whittaker functional on a cuspidal representation and the 
coefficient of proportionality D(s, </>, (p) (i.e., essentially the Rankin-Selberg L-function). In the argument 
we presented, the coefficient of proportionality a(s) in addition depends on the choice of the auxiliary 
model trilinear functional. One can use the Whittaker functional in order to define the model trilinear 
functional and hence eliminate this extra indeterminacy. We hope to return to this subject elsewhere. 

3.6. Proof of Theorem 11.31 We are interested in getting a bound for the coefficients a n (<f>). The idea 
is to find a test vector w 6 V ® V, i.e., a function w S C°°(M x M), such that when substituted in the 
Rankin-Selberg formula (13. 6|) will produce a weight w which is not too small for a given n (\n\ — > oo). 
Given such a vector we have to estimate its spectral density, i.e., the transform w b . One might be tempted 
to take w such that w is essentially a delta function (i.e., the weight w picks up just a few coefficient 
a n (4>) in (|3.6[) ). However, for such a vector we have no means to estimate the right hand side of the 
Rankin-Selberg formula (|3.6[) because the weight function w b is spread over an interval of the spectrum 
which is too long to use the maximum modulus bound (still, conjecturally even for such functions the 
contribution on the right hand side of the Rankin- Sclbcrg formula is small thanks to cancellations). The 
solution to this problem is well-known in harmonic analysis. One takes a function which produces a 
weighted sum of the coefficients |aj;(</>)| 2 in a certain range depending on n and such that its transform 
w spreads over a shorter interval. For a certain kind of such test vectors w (namely, those for which the 
support of w is not too small), we give essentially a sharp bound for the value of Iam{w)- 

We now explain how to choose the required test vectors. Let x be a smooth function on M with the 
support supp(x) C [—^,5] and such that the Fourier transform satisfies |x(£)| > I for |£| < 1. We 
consider the convolution tp = \ * x'i where x'( x ) — x(~ x )- We have supp(ip) C [—1, 1], > for all 
£ and > I for \£\ < I. 

Let N > T > 1 be two real numbers. We consider the following test vector 

w N}T (x, y) = T ■ e-^*^) . ^(T(x - y)) ■ + y) . 

We have the following technical lemma describing properties of w b N T (where the transform b was defined 
in formula (|3.7p and is essentially the Mellin transform in \x — y\). 

Lemma. For u>n,t o,s above, the following bounds hold: 
(1) 1/ w N , T (t,t)dt\ < cT, 
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(2) w N , T (S,-O>0 for allS, 

(3) w N , T (€, -£) > 1 for all £ such that |£ - N\ < T, 

(4) \w% T (s)\ < cT\N\-? for \s\ < N/T, 

(5) \w b NT (s)\ < cT(l + | S |)- 3 for \s\ > N/T, 

for some fixed constant c > which is independent of N and T. 

Bounds (l)-(3) are immediate. Bounds (4) and (5) are standard, once we apply the stationary 
phase method or the van der Corput lemma to the integral of the form w b N T (s) = V>(0) • T^ +s / 2 ■ 
f ip(t) e~ l T' t \t\-i- s / 2 dt (see SectionEU). 



We return to the proof of Theorem 11.41 We will use the following mean value (or convexity) bound 

/ \a(it)\ 2 dt <C T A 2 In A , (3.10) 
Jo 

proved in [BRl] for any A > 1. The constant C T satisfies the bound C T < Cr(l + with a constant 
Cr depending on T only. 

We substitute the vector wn,t into the Rankin-Selberg formula (|3.6p (and use Tr(w) = f w(t,t)dt). 
Taking into account the convexity bound (|3 . 1 0|) , bounds in the lemma and the Cauchy- Schwartz inequal- 
ity, we obtain 

X! k'MI 2 < \ a k{v)\ 2 w N , T {k) = 

\k-N\<T k 

WN,r(t,t) dt + J a(s)w N j-(s) d\s\ < 
<cT+ I cT\N\-i\a(s)\d\s\+ I cT{l + \s\)- 3 \a(s)\ d\s\ < 

J\s\<N/T J\s\>N/T 



<cT + cT\N\-U [ \a{s)\ 2 d\s\- [ 1 d\s\ j + 

\J\s\<N/T J\s\<N/T J 

+ cT f (l + | S |)- 3 (l + | a ( S )| 2 )d| S |< 

J\s\>N/T 
f N\ 3/2+£ 

< cT + CT\N\-% (— J + DT = c'T + CT-?- £ \N\ 1+£ , 

for any e > and some constants c', C, D > 0. 
Setting T = TV 2 / 3 , we obtain £ |flfe(^)| 2 < A £ N 2 ' 3+e for any e > 0. □ 

\k-N\<N 2 /3 



3.7. Remarks. 1. It is more customary to use the formula (13. 9p . We find the geometric formula (|3.6|) 
more transparent. Following the argument of Good |Goj . one usually argues as follows. For R > 1 and 
Z > 1, choose a test function az,n(t) = az(t/R), where az is smooth, supported in (1 — 2/Z, 1 + 2/Z) 
and a\{i-\/z,i+i/z) = 1- This means that the sum \ak{v)\ 2 otz,ii(k) is essentially over k in the interval 
of size R/Z centered at R. The Mellin transform M(az)(s) = L + az{t)\t\ s d x t of az satisfies the simple 
bound 

\M(a z )(s)\ <cZ- x 

for any |s|, and the bound 

\M(a z )(s)\<c\s\- 1 U 
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for any m > and \s\ > 1. This follows easily from integration by parts (we are only interested in 
s € iM.). In particular, we have |M(az)(s)| < cZi +e \s\~ 3 / 2 ~ 6 for \s\ > Z. Using the average bound 
JJ 4 \a{it)\ 2 dt < C T A 2 In A and the Cauchy-Schwartz inequality, one obtains 



a(s)j(s)M(a z ,R){s)ds 



< C E R? +£ Z2 +£ 



for any e > 0. We arrive at J^k \ a k{v)\ 2 a z ,R{k) < R/Z + G E Rh +€ Z^ +e and setting Z = R 1/3 , we obtain 
the bound claimed. 

2. One might conjecture that for any A > 1, the bound 

\a(it)\ 2 dt^ E A 1+£ (3.11) 

A 

holds for any e > (e.g., the Lindcloff conjecture on the average for the Rankin-Selberg L-function). It 
is easy to see that such a bound would lead to the bound |a„(^)| <d, l£ |n|4 +e . We note that this bound is 
a natural barrier which for the Rankin-Selberg method would be hard to overcome. Nevertheless, it was 
suggested by Sarnak |Sa| that for a general lattice T C PGL2(R) the Ramanujan-Petersson conjecture 
\a n (4> T )\ < \n\ E might hold. 



3.8. Proof of Lemma 13.61 We prove the following statement from which Lemma 13.61 immediately 
follows. 

Lemma. Let ip be a fixed smooth function with a compact support in [—1, 1]. For s G it and £ G K, 
denote i/r(£, s) — f R ip(t)e~ l Z t \t\~2~ s dt. There exists a constant c > such that 

(1) |^ b (£,s)| <c(l + K|)-* for\s\<m, 

(2) \^(ti,s)\ < C (l + N)- 3 / r| S |>2|£|. 

In fact, both claims in the lemma are simple consequences of the van der Corput lemma (see [St], 
p. 332). One also can use the following Fourier transform argument. 

To prove (1), we use the fact that the Fourier transform of is equal to — ~^ +s , 

where 7(s) is defined in Remark 13.5.11 and |7(— \ — it)\ = 1. The Fourier transform of ip satisfies 
\tp(0\ *C (1 + |CI) M f° r an y M > 0. Hence, the Fourier transform of i[i(t)\t\~^~ s , i.e., the convolution 
V'(C) * I£I - ^ +S j is bounded by c(l + |£|)~^ for some c and all s E «K. This proves (1). 

To prove (2), it is enough to notice that under the condition |s| > |£ | the phase in the oscillating integral 
defining fi b (^s) has no stationary points. The resulting bound easily follows from the integration by 
parts (see Appendix \X\ for the similar computation). □ 



4. Spherical Fourier coefficients 



When dealing with the spherical Fourier coefficients we assume, for simplicity, that the lattice T is co- 
compact. A general finite co- volume lattice could be treated analogously without any significant changes 
(see Remark |4~5|) . 



4.1. Geodesic circles. We start with the geometric origin of the spherical Fourier coefficients. We fix 
a maximal compact subgroup K — PO(2) C G and the identification G/K — > H, g i— > g ■ i. Let y G Y 
be a point and p:H— >r\H~y the corresponding projection compatible with the distance function 
d(-, ■) on Y and on H. Let R y > be the injectivity radius of Y at y. For any r < R y we define the 
geodesic circle of radius r centered at y to be the set a(r,y) = {y' G Y\A(y' ,y) = r}. Since the map p 
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is a local isometry, we have that p(<Jw{ r , z )) = o~(r, y) for any z G H such that p(z) — y, where cr^(r, z) 
is the corresponding geodesic circle in H (all geodesic circles in HI are the Euclidian circles, though with 
a center different from z). We associate to any such circle on Y an orbit of a compact subgroup on X. 
Namely, let Kq — PSO{2) C K be the connected component of K. Any geodesic circle on H is of the 
form <7jg(r, z) = hK$g ■ i with h, g S G such that h-i = z and hg ■ i £ <j^(r,z) (i.e. an ^.-translation 
of a standard geodesic circle centered at i G H and passing through g ■ i € HI). Note, that the radius of 
the circle is given by the distance d(i,g ■ i) and hence g g 1 K$ for a nontrivial circle. Given the geodesic 
circle a(r, y) C Y, we choose a circle aa(r, z) C H projecting onto cr(r, j/) and the corresponding elements 
5, /i G G. We denote by if a = g~ 1 Kgg the corresponding compact subgroup and consider its orbit 
Kcr = hg ■ K a C X. Clearly we have p(/C<j) = a. We endow the orbit K a with the unique if^-invariant 
measure d[i^ a of the total mass one (from the geometric point of view a more natural measure would be 
the length of a). 

We note that in what follows the restriction r < R y is not essential. From now on we assume that 
K, C X is a connected orbit of a connected compact subgroup K' C G (i.e., K' is conjugated to PSO{2)). 
The restriction r < R y simply means that the projection p(/C) C F is a smooth non-self intersecting curve 
on Y. We also remark that it is well-known that polar geodesic coordinates (r, 8) centered at a point 
zq G H = G/K could be obtained from the Cartan JTAif-decomposition of G (see [He]). This allows one 
to give a purely geometric construction of the functions P„ !T from the Introduction (see Section 14. 2. ip . 

4.2. if'-equivariant functionals. We fix a point 6 G /C. Let x '■ K' ^ S 1 be a character. To such a 
character we associate a function x.(ofc') = x(fc'), k' G X' on the orbit /C and the corresponding functional 
on C°°(X) given by 

<$£(/)= / fWxMdnK 

JK 

for any / G C°°(X). 

The functional d££ is x-equivariant: d™j~(R(k')f) = x(fc') ■ ^x(/) for an y fc ' e where ^ is the 
right action of G on the space of functions on X. For a given orbit K, and a choice of a generator xi 
of the cyclic group K' ~ Z of characters of the compact group if', we will use the shorthand notation 
d^ ut — d^fc, where Xn — X%- The functions Xn- , n <E Z form an orthonormal basis for the space 
L 2 (K,,diJ,!c) (since we normalized the measure by /ijc(/C) = 1). 

Let v : V — > G°°(X) be an irreducible automorphic representation. When it does not lead to confusion, 
we denote by the same letter the functional d°^^ — d°^^ v on the space V induced by the functional 
d"^ defined above on the space C°°(X). Hence we obtain an element in the period space Vk'(V,x) — 
Hom^-'(V, x)- We next use the well-known fact that this space is at most one-dimensional. 

Let V ~ V T be a representation of the generalized principal series. We have then dim Hom^/ (V T , x) < 1 
for any character x of K' (i.e., the space of -fT'-types is at most one dimensional for a maximal connected 
compact subgroup of G). In fact, dimHom^'(VV,Xn) = X if and only if n is even. 

To construct a model x-equivariant functional on V T , we consider the circle model V T ~ G^ en (S 1 ) in 
the space of even functions on 5* 1 and the standard vectors (exponents) e n = exp(in6>) G C°°(S 1 ) which 
form a basis of if-types for the standard maximal compact subgroup K = PO(2). For any n such that 
dimRouiK (V T , Xn) = 1> the vector e' n = ir T (g~ 1 )e n defines a non-zero (xn, ^ftr' ) -equivariant functional on 
V T by the formula 

c od w = c°>) = <".<> • 

We call such a functional the model Xn-equivariant functional on V ~ V T . 



RANKIN-SELBERG WITHOUT UNFOLDING 



25 



The uniqueness principle implies that there exists a constant b n (v) = b Xn jc(v) such that for any v £ V 

dl ut (v)=b n (v)-C° d (v) . 

4.2.1. Functions P n ,r- We want to compare the coefficients b n {v) to the coefficients b n (<f> r ) we introduced 
in (jl.lOp . In particular we describe the functions P n ,T and their normalization. Let h, g £ G and 
K, = hgK' C r \ G = X be the orbit of the connected compact group K 1 — g^K^g as above. Let 
v : V T — > C°°(X) be an automorphic realization and 4> T = v(eo) € C°°(X) the if-invariant vector 
which corresponds to a A"-invariant vector eo £ V T of norm one, i.e., T is a Maass form. We define 
the function P„. r through the following matrix coefficient: Pn^i^e" 10 = (eo,ir T (g~ 1 k~ 1 )e n )v T , where 
(r, 6) — z — /ifcg ■! el for fc £ Kq. It is well-known that the matrix coefficient is an eigenfunction of 
the Casimir operator and hence P„ jT (r)e™ 6 ' is an eigenfunction of A on H. In fact, the functions P n T 
are equal to the Legendre functions for the special choice of parameters (compare to [Iwj ) . Under such 
normalization of functions P n ,r, we have 

\b„(y)\ = \b n {cj) T )\ . 

Let V be the complex conjugate representation; it is also an automorphic representation with the 
realization v : V —> C°°(X). We only consider the case of representations of the principal series, i.e., we 
assume that V — V T , V = V- T for some r € iM.; the case of representations of the complementary series 
can be treated similarly. Let {e rl }„ e 2Z be a K -type orthonormal basis in V . We denote by {e„} the 
complex conjugate basis in V and denote by J°"*/ mod ^ ne corresponding automorphic/model functionals 
on the conjugate space V ~ V- T . 

We introduce another notation for a K' -invariant functional on an irreducible automorphic represen- 
tation Vi : V\; — > C°°(X) of class one. 

Let xo : K' —* 1 G S 1 c C be the trivial character of K' . We have as above 

d WO.( w ) = / v i(v){k)xo.(k)dii K = b (ui) ■ (v,e' ) v , 
Jk 

for any v £ V\ i . 

We denote by d\(v) = (v, e' )v x the corresponding model functional and by 

/3(\i) = b { Vi ) 

the proportionality coefficient (somewhat abusing notations, since the coefficient depends on the auto- 
morphic realization Vi and not only on the isomorphism class Vx 4 ) . 

We want to compare coefficients /3(Aj) with more familiar quantities. Let K, — xqK' C X be an orbit 
of the compact group K' . Let Vi : V\ i — > C°°(X) be an automorphic realization and 4>' x . = Vi{e' ) the 
if '-invariant vector which corresponds to a if'-invariant vector e' € V\i of norm one. From the definition 
of bo{vi) it follows that 

(3(Xi) =d>' Xi (x Q ) . (4.1) 

Hence, under the normalization we choose, the coefficients /3(\i) coincide with the value at a point xq for 
Maass form <f>' x . on the Riemann surface Y' = T\ G/ g~ 1 Kg. 

Finally, we note that on the discrete series representations any if'-invariant functional is identically 
zero. This greatly simplifies the technicalities in what follows. 
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4.3. A/C-restriction. Let A/C C AA C A x A be the diagonal copy of the cycle JC. We define the 
A-fT'-invariant automorphic functional d^K '■ E — V ®V — > Cby 

d&K.{w) = / V E (w)(k,k)dlAK. 
JAK 

for any w £ E. 

Arguing as in Section [3. 1[ we have the following Plancherel formula on K, 

d A K(w)=J2< Ut ® d ' a _ U *(w) = 

(4.2) 

= \ b n(v)\ 2 dT d ® d-nH = E K{v)?w(n, -n) , 

n n 

where w(n, —n) = (w, e n (8> e_ n ) Hence, choosing various vectors w; we can obtain a variety of weighted 
sums J2n \K(v)\ 2 a{n). 

We now obtain another expression for the functional d^fc using the spectral decomposition of L 2 {X) 
and trilinear invariant functionals as in Section [3.41 



4.4. Anisotropic Rankin-Selberg formula. Proof of Theorem 11.41 We assume that the space 
X is compact. This implies the discrete sum decomposition L' 2 (X) = (©j(Lj, z/j)) © {® K {L K , v K )) into 
irreducible unitary representations of G, where Vi : Li ~ L\ t — > L 2 (X) are unitary representations of 
class one (i.e., those which correspond to Maass forms on Y) and L K are representations of discrete series 
(i.e., those which correspond to holomorphic forms on Y). We fix such a decomposition and denote by 
Vi C Li the corresponding spaces of smooth vectors, and by v* : G°°(A) — > Vi the adjoint map. 

We fix v : V — > G°°(A) an irreducible automorphic representation and denote by Ve = v ® v ■ E = 
V <g> V — * C°°(X x X) the corresponding realization of E. 

We use the notations from Section [5~4l Let ta ■ C°°(X x A) — > C°°(A) be the map induced by the 
imbedding A : A > A x A. Let v\ : V\ t — ► C°°(A) be an irreducible automorphic representation. 
Composing ta with the adjoint map v* : C°°(A) — > VX 4 5 we obtain the trilinear AG-invariant map 
T™ 1 : E — ► Va 4 and the corresponding automorphic trilinear functional i* u< on ® V?. defined by 
lf ut (v ® it ® to) = (rA{vE{u ® v)),Di(w)) (we identified Va 4 with the smooth part of V^ 1 .). Such a 
functional is clearly G-invariant, and hence we can invoke the uniqueness principle for trilinear functionals 
from Section l3~4l 

To this end, we fix a model trilinear functional l™ od = l^^y* (see Section 13.51 and the formula (|4. 5[) 

below; for a detailed discussion, see |BR3 ) and the corresponding intertwining model map T\ i = T™ od : 
E -> F A .. This gives rise to the coefficient of proportionality which we denote by <x ( ) — (iv E % y i 
(somewhat abusing notations by suppressing the dependence on ve and v i) such that 

Tr t = v*{TA) = a{\ i )-T Xi . 

The integral pic(f) — Xc f{k)dfJ>K defines the period map pjt : G°°(A) — > C. Note that pic vanishes on 
representations with no non-zero i-T'-invariant vectors, e.g., on representations of discrete series. 

We have the basic relation 

dAK = (»"a)*(p/c) • 

This means that for any w £ E, we have dAfc(w) = L. TA{vE{wy)dpkK. ■ We also have the following 
spectral decomposition for any w £ £7: 

r A (w) = X>*( r A(™)) > (4-3) 
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where Vi runs through the fixed decomposition of L 2 (X) into irreducible components. 

We apply the functional p/c to each term in (I4.3[) and invoke the uniqueness principle for i-C '-invariant 
functionals on irreducible representations V Xi (i.e., that d^ bt = (3(\i) ■ d Xi ; see Section l4.2.ip . This, 
together with the Fourier expansion (|4.2|) , imply two different expansions for the functional c?aac : on e 
which is "geometric" (i.e., the Fourier expansion (|4.2[) along the orbit K.) and another one which is 
"spectral" (i.e., induced by the trilinear invariant functionals and the expansion (|4.3p ). 

Namely, we have 

Y,\bn(v)\ 2 w(n,~n) = d AIC (w) = ^ a(A 4 )/3(A l ) • d Xi (T Xi (w)) , (4-4) 

n Xi 

where w(n,—n) = (w,e' n g) e'_ n )E for any w 6 E, with {e' n } a basis of if'-types in V and {e' n } the 
conjugate basis in V. 

This is our substitute for the Rankin-Selberg formula in the anisotropic case. 

To make this formula explicit, we describe the model trilinear functional in the circle model of repre- 
sentations V — V T , V — V- T and V Xi . We assume for simplicity that r e iK (i.e., V is a representation 
of the principal series) and that there is no exceptional spectrum for the lattice T (i.e., that A^ £ iK for 
all i > 0, and hence V x * ~ V-\). The general case could be treated analogously. 

To simplify formulas, we make the following remark. The formula (|4.4[) appeals only to automorphic 
representations of G and a choice of a (non-trivial) connected compact subgroup K' C G (i.e., the choice 
of another compact subgroup K we made in Section [4.1l is irrelevant). Since there is no preferred compact 
subgroup in G, we may assume without loss of generality that K 1 = PSO(2) is the standard connected 
compact subgroup of G. 

It was shown in |BR3( that in the circle model of class one representations, the kernel of on the 

space E<S>V-x ~ Cj^ en (5 1 x S 1 x S 1 ) is given by the following function in three variables 9, 9', 9" 6 S 1 

K Tt - TiX (d,6',6") = 

\sm(9 - 9')\^\sm(9 - 9")\^^\ S m(0' - O'^l" 11 ^ ■ ' 4 "" 



This also defines the kernel of the map T\ : E — * V\ via the relation 

f 

(2^) 3 V)» 



(T A (w),-y}y A = t^-^ / w(9,O')v{6')K T - T ,x(6,0',6")d9d9'd9" 



We have d x {T x (w)) = (T x (w),e ) Vx = ^ / w(0,9')K r - T , x (9,9' ,9")d9d9'd6" for any vector w € 
C 00 (5 1 xS 1 ). It is clear from the formula flUD that we can assume without loss of generality that 
the vector w G E is Aif-invariant. Such a vector w can be described by a function of one variable; 
namely, we set w(9,9') = u(c) for u £ C°°(S 1 ) and c = (9 — 9')/2. We have then w(n, —n) = u(n) = 
25jr Jci u(c)e~ mc dc, i.e., u is the Fourier transform of it. 

We introduce a new kernel 

k x (c) = fc r , A (*#) = ^ / K T ^ T , X (9, 9', 0")d9" (4.6) 
and the corresponding integral transform 

tt»(A) = u\{\) = f u(c)k x (c)dc , (4.7) 



(2^) 2 



s 1 
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suppressing the dependence on r as we have fixed the Maass form ef> r . The transform is clearly defined 
for any smooth function u E C°°(S 1 ), at least for A E iR. In fact, it could be defined for all A E C, by 
means of analytic continuation, but we will not need this. 

Note that k\ is the average of the kernel K Tt - Ti \ with respect to the action of AK, or, in other terms, 
is the pullback of the if-invariant vector eo E V\ under the map i.e., k\ — T£(eo) E E*. We 
also note that the contribution in (|4.4j) coming from the trivial representation (i.e., A = 1) is equal to 
u(0) = vol ^ i . u{0) under our normalization of measures vol(X) — volQC) = 1. 

vol(X)? 

The formula (|4.4p then takes the form 

\b n {v)\ 2 u(n) = u(0) + a(Xi)0(Xi) ■ u^X,) . (4.8) 

This formula is an anisotropic analog of the Rankin-Selberg formula (|3.8[) for the unipotent Fourier 
coefficients of Maass forms. We finish the proof of Theorem II. 41 □ 

4.5. Remarks. 1. The kernel function k\ is not an elementary function, unlike in the case of the 
unipotent Fourier coefficients where its analog is given by \x — y\~2~ s . This is related to the fact that 
the iV-invariant distribution S s on V s is also x-equivariant under the action of the full Borel subgroup 
B = AN for an appropriate character x of B which is trivial on N. The space of (B, x)-equivariant 
distributions on E is one-dimensional for a generic x- This is due to the fact that B has one open orbit 
for the diagonal action on the space Mxl and the vector space E is modelled in the space of smooth 
functions on this space. It is easy to write then a non-zero £?-equivariant functional on E by an essentially 
algebraic formula. We do not have a similar phenomenon for a maximal compact subgroup of G. We will 
obtain however, an elementary formula for leading terms in the asymptotic expansion of k\ as |A| — > oo 
(see Appendix lA.ip . 

2. For Hecke-Maass forms, the proportionality coefficient a(s) in the Rankin-Selberg formula (13. 4[) for 
the unipotent Fourier coefficients gives the Rankin-Selberg L-function (after multiplication by ((2s)). 
In the anisotropic case we do not know how to express the coefficient a(Aj) in terms of an appropriate 
L-function. It is known that the value of a(Ai)| 2 is related to the special value of the triple L-function 
(see [W]), but not the coefficient itself. The same is true for the coefficient (3(\i) where in special cases 
\f3(Xi)\ 2 is related to certain automorphic L-function (see [Waj . |JNj ). There still might be a way to 
normalize the product a(Xi)/3(Xi) in a canonical way. We hope to return to this subject elsewhere. 

3. For a non- uniform lattice T , the proof we gave above leads to the following formula analogous 
to (|4.8p which includes the contribution from the Eisenstein series. Namely, for the similarly defined 
coefficients tife(s) and /3fc(s) corresponding to the Eisenstein series contribution, we have 

Y \bn{y)\ 2 u(n) = u(0) + Y «(A J )/3(A l ) • u J (A,) + £ / a k (s)0 k (s) ■ u*(s)ds . 

n Xfjtl cusps "' iR+ 

4.6. Bounds for spherical Fourier coefficients. Proof of Theorem 11.51 We follow the strategy of 
Section l3~6l We construct a Aif-invariant test vector w E V ® V, i.e., a function u E C CC (S 1 ), such that 
when substituted into the Rankin-Selberg formula (|4.8p will produce a weight u which is not too small 
for a given n, \n\ — > oo. We then have to estimate the spectral density of such a vector, i.e., the transform 
it". In fact, as in Section \3l6\ we take a function which produces a weighted sum of the coefficients \bk(4>)\ 2 
for k in a short range depending on n and such that its transform is spread over a relatively short 
range of A's. For such test vectors w we give an essentially sharp bound for the value of cIak(w). 

We have the following technical 
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Lemma. For any integer N and a real number T such that N > T > 1, there exists a smooth function 
t S C°°(S 1 ) satisfying the following bounds: 

(1) \u N , T (0)\ < olT, 

(2) u NiT (k) > for all k, 

(3) UN,r(k) > 1 for all k satisfying \k — N\ <T, 

(4) |u^ iT (A)| < aT\N\-? {1 + \\\)~? + aT{l + \\\- 5 / 2 ) for \\\ <N/T, A £ iM, 

(5) |< T (A)| < qT(1 + |A|)- 5 / 2 /or |A| > JV/T, 

/or some fixed constant a > independent of N and T . 

The proof of this Lemma is given in Appendix A. We construct the corresponding function un,t by 
considering a function ujv,t( c ) = T 2 ■ e~ lNc ■ (ipx * ifj' T ) (c), where * denotes the convolution in C co (S 1 ) 
and tp' T (c) — -0t(— c). Here V't(c) = V* (T ' c )> where ^ £ C°°(S' 1 ) is a /ixed smooth function supported 
in a small neighborhood of 1 G S , and T ■ c means the obvious scaling-up of the angle parameter in S . 
Functions un,t obviously satisfy bounds (l)-(3) and the verification of (4)-(5) is reduced to a routine 
application of the stationary phase method (similar to our computations in |BR4j ) . These bounds are 
similar to bounds in Lemma 13.61 for the unipotent Fourier coefficients. There are two differences though. 
First, in the corresponding bound in (4) there is a factor of (1 + |A|)~ a in the first term. This constitutes 
an important difference between if-invariant and iV-invariant functionals on the representation E. The 
additional term T(l + | A | ) comes from the estimate of the remainder in the stationary phase method 
and could be improved further (although it would not make a difference in what follows). The second 
(minor) difference is that the integral transform ^ is elementary (i.e., the Mellin transform) while the 
integral transform " has its kernel given by a non-elementary function (essentially by the hypergeometric 
function). This slightly complicates computations. 

Remark. We would like to point out that it is absolutely essential that in the integral u^ NT (X) = 
fgi UN,T(c)k\(c) dc the support of the function itjv,T does not contain points ±7r/4, ±37r/4. Otherwise the 
phase in the above oscillating integral possess degenerate critical points at these values of c for N/T X |A|. 
The presence of degenerate critical points change drastically the behavior of the corresponding transform 

In particular, for n x |A|, the jj-transform of a pure tensor e n <8> e_„ does not satisfy the bound (4) in 
the above lemma. Namely, (e„ <g> e_„)"(A) have a sharp peak for n x |A| of the order of |A|~B (as oppose 
to |A| _1 ). This phenomenon is the starting point for the proof of the subconvexity bound for the triple 
L-function given in |BR4| . In the present paper, we choose test vectors to vanish in a neighborhood of 
these degenerate points ±7r/4, ±37r/4 in the model realization V ® V ~ C°°(S 1 xS 1 ). This allows us to 
avoid the more delicate analysis of degenerate critical points. Note that our test vectors are not given by 
a finite combination of pure tensors of K-types. 

We return to the proof of the theorem. In the proof we will use two bounds for the coefficients a(Ai) 
and /3(Aj). Namely, it was shown in [BR3] that 

]T HA,)| 2 <aA 2 , (4.9) 

A<|Ai|<2.4 

for any A > 1 and some explicit a > 0. The second bound we will need is the bound 

£ \P(\)\ 2 < bA 2 , (4.10) 

A<\X t \<2A 

valid for any A > 1 and some b. In a disguise this is the classical bound of L. Hormander |Ho] for the 
average value at a point for eigenfunctions of the Laplace-Beltrami operator on a compact Riemannian 
manifold (e.g., A on Y). This follows from the normalization |/3(Ai)| 2 = |^.(a;o)| 2 we have chosen in 
(|4.1|) for A"'-invariant eigenfunctions. In fact, the bound (|4.10p is standard in the theory of the Selberg 
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trace formula (see [Tw ) and also can be easily deduced from simple geometric considerations of [BR3 . 
We note that both bounds are consistent with the convexity bounds for relevant L-functions. 

We also assume for simplicity that there is no exceptional spectrum (i.e., A; E iM. for i > 0). A general 
case could be treated analogously by extending Lemma [4761 to cover A G (—1,1) or as in [BR4] where we 
treated exceptional spectrum using simple considerations based on the Sobolev restriction theorem. 

We plug a test function satisfying bounds (1) — (5), Lemma 14751 into the Rankin-Selberg formula (|4.8j) . 
From the Cauchy-Schwartz inequality, bounds (|4.9[) and (|4.10p . and summation by parts, we obtain 

Y M^)| 2 < Y \b k {v)\ 2 W(fc) = u N , T (0) + Y «(^)/3(A l ) < )T (Ai) < 

\k-N\<T k A«#l 

<aT+ Y o t T\N\-i(l + \\ i \)-?\a(\ i )f3(\ i )\ + 

\\i\<N/T 

+ Y aT(l + |Ai|)-'|a(A i )/3(A i )| < 
<aT + aT\N\-l. Y (1 + (|«(A,)| 2 + |/3(A,)| 2 ) + 

\Xi\<N/T 

+ r^(i + |A i |)-l(KA l )| 2 + |/3(A l )r) < 

(N\ 
- +DT = c'T + CT-?- e \N\ 1+£ , 

for any e > and some constants c', C, D > 0. 
Setting T = N §, we obtain £ \b k {v)\ 2 < A £ N% +E for any e > 0. □ 

|fc-JV|<AT2/3 

Remark 4.1. One expects that bounds |a(Ai) <C \Xi\ £ and |/3(Ai)| <C |Ai| £ hold for any e > 0. In special 
cases this would be consistent with the Lindeloff conjecture for the corresponding L-functions. This 
however, will not have the similar effect on the bound in Theorem 11.51 for spherical Fourier coefficients 
b n ((j) T ). The reason for such a discrepancy is that the spectral measure of the Eisenstein series is much 
"smaller" than that of the cuspidal spectrum. Nevertheless, it is natural to conjecture that for general 
r C PGL2(R) and a point yo G Y the spherical Fourier coefficients satisfy the bound \b n ((f> T )\ -C \n\ £ . 
For a CM-point yo and a Hecke-Maass form, this would correspond to the Lindeloff conjecture for the 
special value of the corresponding L-function via the Waldspurger formula () 1 . 1 2|) . 



Appendix A. Asymptotic expansions 

A.l. Asymptotic expansion for the kernel k\. We set c = and consider the integral (|4.6|) , 

Section [121 

kx{c) = k T , x = i / K T r \ {6 , 9' , 6") d6" = 

27T J S 1 

= — . |sin(2c)|-5-7 • / \sm{0" -c)\-?- T+ ?\sm(6" + c)\-? +T+ id6" 

27T Jgl 

= \sm(2c)\-7-*KxAc) . 
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where the kernel K_ T T \ is as in (|4.5p and we denoted by K\ T the integral 

K Xt {c) = — - [ |sin(i-c)|-^+t|sin(* + c)|-5+ T+ ^ . (A.l) 
2vr J s i 

The kernel K\ T is not given by an elementary function. We obtain an asymptotic formula for K \ T by 
applying the stationary phase method to the integral (|A.1[) . The asymptotic formula we obtain is valid 
for a fixed r and is uniform for A 6 iM. and c ^ 0, ±7r/2, it. We denote the set of exceptional points by 
S = {0, ±7r/2, it} C 5 1 . These points are singular because the integrand in (|A.1[) degenerates for these 
values of c. We have the following 

Claim. There are constants A, B and C such that for all A 6 iM. and c S , 

K\. T {c) = m A (c) + m A (c + 7r/2) + r T (A, c) , 

where the main term m\(c) is a smooth function of X and c (c$ S), and for \X\ > 1 is given by 

m x (c) = |A|"5 (A + BlAr 1 +C|A|- 1 cos 2 (c)) • |sin(c)| A . 

The remainder r T (X,c) satisfies the estimate 

|ry(A, c)| = O ((1 + |A|)- 5 / 2 + [1 + |ln(| sin(c) cos(c)|)|] • (1 + |A|)- 10 ) 
with the implied constant in the O-term depending on r only. 
Hence we have 

k x (c) = \ S m(2c)\-^-^KxAc) = 

M x (c) + M x (c + tt/2)+| sin(2c)|-*-*r r (A,c) , 
with M x (c) = \X\~i [A + BIAI- 1 +C|A|- 1 cos 2 (c)] • | sin(2c)|-i| sin(c)|^| cos(c)|"3. 

A. 2. Proof. The asymptotic expansion in the claim follows from the stationary phase method applied 
to the integral (|A. 1[) . We consider the asymptotic expansion consisting of two terms and a reminder. 
Since all functions are 7r-periodic we consider only the interval c e [0, n]. For c ^ S, the phase of the 
oscillating kernel in the integral (jA.lj) has two non-degenerate critical points at t = and t = ir/2. 
Hence, the asymptotic expansion is given by the sum of two terms. It turns out that these terms have 
the form m x (c) and m\(c + tt/2) for the same function m\. Singularities of the amplitude at t = c, tt — c 
are responsible for the logarithmic term in the remainder. For |A| — > oo, this contribution from the 
singularities of the amplitude is of order of 0((1 + |A|)~ fc ) for any k > due to the fast oscillation of 
the phase at the same points. In fact, in |BR4j we gave a self-contained treatment of such (and more 
complicated) integrals based on the reduction to standard integrals and the use of the van der Corput 
lemma. Here we show how one can deduce necessary bounds from the stationary phase method. 

Our computations are based on the following well-known form of the two-term asymptotic in the 
stationary phase method (see |Bo| . [F] , [St]). We also use the estimation of the corresponding remainder. 

Let 4> an( i / be smooth real valued functions on S . To state the stationary phase formula, we assume 
that <j> has a unique non-degenerate critical point to € S 1 . We consider the integral /(A) = f sl f{t)e x ^^dt 
for A G iM.. For |A| > 1, we have the following expansion 

I(X) = |A|-5(C + C 1 |Ar 1 )e A ^)+r(A) , (A.6) 
where C = (27r)5e l - si s n ^"( t °)) 7r / 4 |0"(i o )|-2/(i o ) an d 

Ci = (tt/2)' e 3i-sign(0"(t o )W4 |^"(£ )|-f. 

•[/" - <M 3) W~0 (4) W + 5(^) 2 //12(0") 2 ] t=to , 



(A.2) 
(A.3) 
(A.4) 
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and the remainder satisfies r(A) = 0((1 + |A|) -5 / 2 ). The constant in the O-term is bounded for <fi and 
/ in a bounded, with respect to natural semi- norms, set in C°° (5* 1 ). If <f> has a number of isolated non- 
degenerate critical points then the asymptotic is given by the sum over these points of the corresponding 
contributions. 

For |A| < 1, we have the trivial bound: |/(A)| < / |/| dt. 

A.2.1. Leading terms. We apply these formulas to compute leading terms in the asymptotic expansion 
of the integral (|A.1[) . We set 

(j){t) = ln|sin(t- c)| + In | sin(< + c)\ and f(t) = | sm{t ~ c)\~?~ T \ sin(f + c)|~i +T . 

We have 4>'(t) — sin(2t)/ sm(t — c) sm(t + c) and hence the phase <f> has two critical points t = and 
t = 7r/2, assuming that c ^ 0, 7r/2, tt. 

A straightforward computation gives for t = 0, 

<£"(0) = -2sin- 2 (c), </> (3) (0) = 0, (4) (O) = -4(1 + 2 cos 2 (c))/ sin 4 (c) 

and /(0) = | sin(c)!- 1 , /"(0) = | sin(c)|- 3 (l + 4r 2 cos 2 (c)) , and similarly for t = tt/2, 

0"(tt/2) = -2cos- 2 (c), (3) (tt/2) =0, = -4(1 + 2 sin 2 (c))/ cos 4 (c) 

and /(tt/2) = |cos(c)|"\ /"(tt/2) = | cos(c)|" 3 (l + 4t 2 sin 2 (c)) . 

Plugging this into (|A.6|) we see that for c ^ 0, ±tt/2, 7T, 

#a,t(c) = m A (c) + m A (c + tt/2) + r(A, c) , (A.7) 

where m\{c) = |A| - ^ (A + S|A| _1 + C|A| _1 cos 2 (c)) ■ | sin(c)| A with some explicit constants A, B, C. 

A. 2. 2. The remainder. We need to estimate the remainder r(A, c) = r T (A,c) as c keeps away from the 
singular set S C S 1 . We claim that 

|r T (A, c)\ = O ((1 + |A|)- 5 / 2 + [1 + | ln(| sin(c) cos(c)|)|] • (1 + |A|)- 10 ) . 

To see this we first note that the contribution to the remainder coming from the integration over any fixed 
interval which does not include singularities of the phase and of the amplitude (i.e., points t = ±c, 7r±c) 
is of order 0((1 + |A|)~ 5 / 2 ) (with the constant in the O-term depending on the proximity of the interval 
to these singular points and on r). 

The analysis of the integral is identical for all points in S, hence we treat only the case of < c < j. 

We use the appropriate partition of unity in order to separate different behavior of the kernel. Let 
io, i c , i- Cl i n /2, i-7r/2j iir-c, in+c, *ir C S 1 = R/27rZ be the following collection of closed overlapping inter- 
vals: io = [— 3c/4, 3c/4], i c = [c/4, 7r/3], i- c = [— tt/3, -c/4], i„ = tt + io, i v - c = n + i- c , i^ +c — n + i c 
and similarly i^ /2 = [tt/4, 3tt/4], i^^ j2 = [— tt/4, -3tt/4]. Let 1 = xo + Xc + X-c + Xtt/2 + X-tt/2 + 
Xtt-c + Xtt+c + Xtt be the corresponding partition of the unity on S 1 separating singular points ±c, n ± c, 
from the stationary points 0, tt, ±tt/2 (i.e., supp(xo) C io etc.). For Xi as above, we denote by I\ T the 
corresponding integral 

4,r(c) = ^-- / \ S in(t-c)\-^^\sm(t + c)\-i + ^ Xt (t)dt . 

27T J s i 

We have K\ tT (c) —J2^\t( c )- -Due to the symmetry, it is enough to deal with the integrals I® T ,I% T ,I^. 

The integral J^ 2 (c) falls under the standard stationary phase method and hence the remainder in the 
two-term asymptotic is of order of 0([1 + |A|]~ 5 / 2 ) (with the constant in the O-term depending on r). 
In fact the behavior of the integral I® T (c) is similar. This could be seen easily by scaling-up the variable 
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t by c. In particular, integrals I x (c) and J A '^(c) give rise to the leading terms in the asymptotic in 
Claim lATTI and the remainder which is of the order of 0([1 + |A|] -5 / 2 ). 

The behavior of the remaining integral 1^ T is similar to the well-known Beta type integral of the form 

B(X, x) — J \ x + l|~^~ r+ ^ \x — 1\~^ +T+ ^ xi x ) dx for a compactly supported smooth function \, vanishing 
in a neighborhood of the stationary point (i.e., near x — 0). (In fact, in |BR4j we showed how to reduce the 
integral 1^ to the integral B(X, x) using an appropriate change of variable.) It follows from integration 
by parts that for such x, the integral B(X, x) is of the order of 0([1 + A|]~ fe ) for any k > 0. The similar 
analysis is applicable to the integral 1^ T . 

For |A| < 1, the integral 1^ T is trivially of the order of 0{\ ln(| sin(c) cos(c)|)|). 

For A | > 1 and small c, we consider two integrals 

^) = f ' f \sHt-c)\^- T+ ^\sm(t + c)\-^H Xc{t )dt , 

Z7r Jc/i 

J+ (c) = ^-- / ' \sm(t-c)\- 1 2--H\ s i n (t + c)\-i+ T +i X c(t)dt . 

27r Jc 

Scaling- up the variable t by c, we see that the integral J^~ T (c) transforms into an integral of the form 
Ji l/( x )l ~^ +x ^ 2 ip(x), where f(x) = f c (x) is a monotone smooth function with derivatives bounded 

4 

for x G [1/4, 1], uniformly in c, and satisfying /(l) = and f'(x) > 1 for x G [1/4, 1]. The function ip 
is a smooth function also with, uniformly in c, bounded derivatives (depending on r). Hence integration 
by parts implies that the integral J^ T (c) is of the order of 0([1 + |A|]~ fe ) for any k > with the constant 
in the O-term independent of c. 

Scaling-up the variable t by c, we see that the integral J^" r (c) transforms into an integral of the form 

J^" \(x — 1) ■ g(x)\~^ +x / 2 ip(x), where g is a monotone smooth function with, uniformly in c, bounded 
derivatives, satisfying g(l) — 1 and g'(x) > 1 for x G [l,7r/3c], and ijj is a smooth function with, 
uniformly in c, bounded derivatives (depending on r). To estimate such an integral, one breaks the 
interval [1, 7r/3c] into | hi(c)| dyadic intervals. On each such interval we have the bound as above of the 
order of 0([1 + |A|] _fe ) for any k > with the constant in the O-term independent of c. Hence the integral 
J+ T (c) is of the order of 0{\ ln(c)| ■ [1 + \\\]- k ) for any k > 0. □ 

A. 3. Proof of Lemma l4.6l We have to analyze the integral transform given by u^ N T ( A) = J UN,T{c)k\(c)dc, 
where u NtT (c) = T 2 ■ e~ iNc ■ (ip T *ip' T ) (c) with the parameters iV > T > 1, N G Z, T G M. Here 
■ijj G C°°(S 1 ) is a fixed smooth function of a support in a small interval containing 1 G S , and 

i/j t (c) = ip(T ■ c), ip' T (c) = i> T (-c). 

To analyze the asymptotic of u^ N T (A), we consider the following model integral 
I(X,N,T)=T J e- lArc |sin(2c)|-^|sin(c)|7|cos(c)|-*x(rc) dc , 

where x is a smooth function with supp(x) C [—1, 1]. 

On the basis of the asymptotic expansion (|A.5|) for the kernel k\, we see that u NT (X) is of the order 
of /(A, N,T) ■ (1 + \X\)~i + 0(T(1 + |A|)- 5 / 2 ). 

We claim that for |A| < N/T, we have \I(X,N,T)\ = 0(TN~*) and for |A| > N/T, we have 
\I(X, N,T)\ = 0(|A| _fe ) for any k > 0. These bounds imply the claim in Lemma T4. 6 1 

To obtain desired bounds for /(A, N, T), we appeal again to the stationary phase method. 
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Namely, scaling-up by T the variable c in the integral /(A, N, T), we arrive at the integral 
h(X,N,T) = J e - ^'|sin(£i)|~*|taji(£)|*x(*)<ft . 

It is easy to see that for |A| < 1, this integral is of the same order as the integral J \t\~^ + ^ e~ l T rt \(t)dt, 
which is of the order of 0(TN~i). For 1 < |A| < N/T, the phase function in the integral Ii has unique 
non-degenerate critical point and the contribution from the singularities of the amplitude is negligible. 
Hence, arguing as in Section [A. 2. 21 we see that the integral I\ is of the order of 0{TN~^). 

In fact, both cases follow immediately from the van der Corput lemma (see jBR4j for similar bounds). 

For |A| > N/T, the phase function has no critical points and hence we have = 0([1 + |A|] _fe ) for 
any k > 0. □ 



References 

[Be] J. Bernstein, Eisenstein series, lecture notes. Park City, Utah (2004). 

[BR1] J. Bernstein and A. Reznikov, Analytic continuation of representations and estimates of automorphic forms, Ann. of 
Math. (2) 150 (1999), no. 1, 329-352. MR1715328 (2001h:11053), arXiv: |math.RT/9907202| 

[BR2] , Sobolev norms of automorphic functional, Int. Math. Res. Not. 2002, no. 40, 2155-2174. MR1930758 

(2003h: 11058) 

[BR3] , Estimates of auto morphic functions, Mosc. Math. J. 4 (2004), no. 1, 19-37, 310. MR2074982 (2005f: 11097), 

arXiv: |math.RT/0305351| 
[BR4] , Subconvexity of triple L-functions, preprint, arXiv: |math.NT/0 608555 (2006). 

[B] A. Borel, Automorphic forms on SL 2 (R), Cambridge Univ. Press, Cambridge, 1997. MR1482800 (98j: 11028) 

[Bo] V. A. Borovikov, Uniform stationary phase method, IEE, London, 1994. MR1325462 (96b:58111) 

[Bu] D. Bump, The Rankin-Selberg method: an introduction and survey, in Automorphic representations, L-functions 

and applications: progress and prospects, 41-73, de Gruyter, Berlin. MR2192819 (2006k: 11097) 
[BM] R. Bruggeman, Y. Motohashi, A new approach to the spectral theory of the fourth moment of the Riemann zeta- 

function. J. Reine Angew. Math. 579 (2005), 75-114. MR2124019 (2005i:11108) 
[F] M. Fedoruk, Asymptotic methods in analysis. Analysis. I, Encyclopaedia Math. Sci., 13, Springer, Berlin, 1989. 

MR1042759 (90j:00027) 

[G5] I. M. Gelfand, M. I. Graev and N. Ya. Vilenkin, Generalized functions. Vol. 5, Academic Press, New York, 1966. 
MR0207913 (34 #7726) 

[G6] I. M. Gelfand, M. I. Graev and I. I. Pyatetskii-Shapiro, Representation theory and automorphic functions, Saunders, 
Philadelphia, 1969. Reprint of the 1969 edition. Generalized Functions, 6. Academic Press, Inc., Boston, MA, 1990. 
MR0233772 (38 #2093). 

[Go] A. Good, Cusp forms and eigenfunctions of the Laplacian, Math. Ann. 255 (1981), no. 4, 523-548. MR0618183 
[Gr] B. H. Gross, Some applications of Gelfand pairs to number theory, Bull. Amer. Math. Soc. (N.S.) 24 (1991), no. 2, 

277-301. MR1074028 (91i: 11055) 
[He] S. Helgason, Groups and geometric analysis, Amer. Math. Soc, Providence, RI, 2000. MR1790156 (2001h:22001) 
[Ho] L. Hormander, The analysis of linear partial differential operators. IV, Springer, Berlin, 1985. MR0781537 

(87d:35002b) 

[Iw] H. Iwanicc, Spectral methods of automorphic forms, Amer. Math. Soc, Providence, RI, 2002. MR1942691 
(2003k:11085) 

[IS] H. Iwaniec and P. Sarnak, Perspectives on the analytic theory of L-functions, Geom. Funct. Anal. 2000, Special 

Volume, Part II, 705-741. MR1826269 (2002b:11117) 
[JN] H. Jacquet and N. Chen, Positivity of quadratic base change L-functions, Bull. Soc. Math. France 129 (2001), no. 1, 

33-90. MR1871978 (2003b: 11048) 
[KSa] H. Kim, P. Sarnak, Appendix to: H. H. Kim, Functoriality for the exterior square of GL4 and the symmetric fourth 

of GL 2 , J. Amer. Math. Soc. 16 (2003), no. 1, 139-183. MR1937203 (2003k:11083) 
[KS] B. Krotz and R,. J. Stanton, Holomorphic extensions of representations. I. Automorphic functions, Ann. of Math. (2) 

159 (2004), no. 2, 641-724. MR2081437 (2005f:22018) 
[Ku] T. Kubota, Elementary theory of Eisenstein series, Kodansha, Tokyo, 1973. MR0429749 (55 #2759) 
[Kz] N. V. Kuznetsov, Sums of Kloosterman sums and the eighth power moment of the Riemann zeta-function, in Number 

theory and related topics (Bombay, 1988), 57-117, Tata Inst. Fund. Res., Bombay. MR1441327 (98c:11085) 
[L] J. B. Lewis, Eigenfunctions on symmetric spaces with distribution-valued boundary forms, J. Funct. Anal. 29 (1978), 

no. 3, 287-307. MR0512246 (80f:43020) 



RANKIN-SELBERG WITHOUT UNFOLDING 



35 



[M] H. Maass, Uber eine neue Art von nichtanalytischen automorphen Funktionen und die Bcstimmung Dirichletscher 

Reihen durch Funktionalgleichungen, Math. Ann. 121 (1949), 141-183. MR0031519 (11,163c) 
[MW] K. Martin, D. Whitehouse, Central L-values and toric periods for GL(2), preprint (2007). 

[Mol] Y. Motohashi, Spectral theory of the Riemann zeta-function, Cambridge Univ. Press, Cambridge, 1997. MR1489236 
(99f:11109) 

[Mo2] , A note on the meanvalue of the zeta and L-functions, preprint, arXiv: |math. NT /0401085 

[O] A. I. Oksak, Trilincar Lorcntz invariant forms, Comm. Math. Phys. 29 (1973), 189-217. MR0340478 (49 #5231) 
[PS] Y. N. Petridis and P. Sarnak, Quantum unique ergodicity for SL2(0)\H 3 and estimates for //-functions, J. Evol. 

Equ. 1 (2001), no. 3, 277-290. MR1861223 (2003a:11060) 
[Pr] D. Prasad, Trilinear forms for representations of GL(2) and local e-factors, Compositio Math. 75 (1990), no. 1, 1-46. 

MR1059954 (91i:22023) 

[Ra] R. A. Rankin, Contributions to the theory of Ramanujan's function r(n) and similar arithmetical functions. II. 
The order of the Fourier coefficients of integral modular forms, Proc. Cambridge Philos. Soc. 35 (1939), 357-372. 
MR0000411 (l,69d) 

[R] A. Rcznikov, Norms of geodesic restrictions for eigenfunctions on hyperbolic surfaces and representation theory, 

preprint (2004). arXiv: |math.AP/0403437| 
[Sa] P. Sarnak, Integrals of products of eigenfunctions, Internat. Math. Res. Notices 1994, no. 6, 251-261. MR1277052 
[Sc] A. Selberg, On the estimation of Fourier coefficients of modular forms, in Proc. Sympos. Pure Math., Vol. VIII, 

1-15, Amer. Math. Soc, Providence, R.I. MR0182610 (32 #93) 
[St] E. M. Stein, Harmonic analysis: real-variable methods, orthogonality, and oscillatory integrals, Princeton Univ. 

Press, Princeton, NJ, 1993. MR1232192 (95c:42002) 
[V] A. Venkatesh, Sparse equidistribution problems, period bounds, and subconvexity, preprint (2005). arXiv: 

|math.NT/0506224"l 

[Wa] J.-L. Waldspurger, Sur les valeurs de certaines fonctions L automorphes en leur centre de symetrie, Compositio 

Math. 54 (1985), no. 2, 173-242. MR0783511 (87g:11061b) 
[W] T. Watson, Thesis, Princeton, 2001. 

Bar-Ilan University, Ramat-Gan, Israel 



E-mail address: reznikovOmath.biu.ac.il 



